Objective Vision-Based Assessment of Parkinsonism and Levodopa-Induced Dyskinesia in Persons with Parkinson s Disease

Transcription

1 Objective Vision-Based Assessment of Parkinsonism and Levodopa-Induced Dyskinesia in Persons with Parkinson s Disease by Michael Li A thesis submitted in conformity with the requirements for the degree of Masters of Applied Science Institute of Biomaterials and Biomedical Engineering University of Toronto Copyright by Michael Li 2017

2 Objective Vision-Based Assessment of Parkinsonism and Levodopa-Induced Dyskinesia in Persons with Parkinson s Disease Abstract Michael Li Masters of Applied Science Institute of Biomaterials and Biomedical Engineering University of Toronto 2017 Parkinson s disease (PD) is the second most common neurodegenerative disease, with the incidence rate climbing rapidly as the global population ages. While levodopa is effective at treating PD symptoms, prolonged usage can introduce additional motor complications called levodopa-induced dyskinesias (LID). Assessment of PD/LID requires regular visits to a clinic; however, the intermittent nature of assessments can fail to capture changes in a person s condition. With computational power becoming increasingly more affordable, computer vision is an accessible means of performing more frequent and objective PD assessments. A study of deep learning for human pose estimation was conducted to examine the feasibility of extracting body movements from videos of PD assessments. Machine learning was applied to detect and estimate the severity of PD/LID using movement features. Results indicate computer vision is a promising candidate for objective PD assessment, thus laying the foundation for an automated system for evaluation of PD motor symptoms. ii

3 Acknowledgments First and foremost, I would like to thank my supervisor Dr. Babak Taati for his guidance and support. Throughout my thesis, he was always encouraging and patient, and somehow managed to balance giving me the freedom to pursue directions of my choosing while reining in my more scattered ideas. To my co-supervisor, Dr. Geoff Fernie, and committee members Dr. David Fleet, Dr. Susan Fox, and Dr. José Zariffa, thank you for your invaluable feedback and for broadening my perspectives. I would also like to thank Dr. Sven Dickinson for serving as the external reviewer for my defense. In addition to serving as members of my committee, I would like to thank David and Susan for lending their time to discuss aspects of my thesis in detail. Thank you to David and Sara Sabour for helping me to set up the deep learning algorithms on a shared server. Thank you to Susan and Dr. Tiago Mestre for their support in analyzing and interpreting the dataset. To the members of the Intelligent Assistive Technology and Systems Lab (IATSL), I would like to express my gratitude for taking care of me for the past few years and nurturing my growth as a researcher and as a person. To Ahmed Ashraf and Shehroz Khan, thank you for the discussions and for your input when I was spinning my wheels. To all the wonderful people that I had the pleasure of knowing while at Toronto Rehab, you have all played an instrumental role in maintaining my sanity over the course of this thesis. To the IATSL leadership Dr. Alex Mihailidis, Dr. Rosalie Wang, and my supervisor Dr. Babak Taati you have assembled a group of kind and talented individuals that I am honoured to have been a part of. A special thank you to my friends (old and new), as well as the Biomedical Engineering Students Association (BESA) and Engineering World Health (EWH) for being a temporary reprieve and helping me recharge the batteries after long days. Finally, thank you to my family for your encouragement and for always pushing me to improve. iii

4 Table of Contents Acknowledgments... iii Table of Contents... iv List of Tables... viii List of Figures...x Acronyms... xii Chapter 1 Introduction Motivation Objectives...3 Chapter 2 Literature Review Parkinson s Disease (PD) and Levodopa-Induced Dyskinesia (LID) Pathophysiology and Clinical Presentation Treatment Options Rating Scales Technologies for PD/LID Assessment Wearable Sensing Vision-Based Other Methods Computer Vision Methods for Pose Estimation...14 Chapter 3 PD Assessment Dataset Overview Study Population Study Protocol...18 iv

5 3.4. Task Selection...19 Chapter 4 Benchmarking Deep Learning Pose Estimation Algorithms Introduction Methods Algorithms Benchmark Evaluation Results Discussion...27 Chapter 5 Feature Exploration Introduction Background Methods Trajectory Extraction Feature Extraction Evaluation Results Communication Drinking Leg Agility Toe Tapping Discussion...52 Chapter 6 Predicting Presence and Severity of PD/LID Introduction Background Machine Learning...58 v

6 Previous Work Methods Classification Regression Results Binary Classification and Regression Multiclass Classification Discussion Performance Analysis Feature Importance Regression Multiclass Classification...83 Chapter 7 Clinical Utility of Objective Features Background Methods Results Discussion...88 Chapter 8 Conclusions and Future Work Objectives Summary of Contributions Limitations Vision Closing...94 References...95 Appendix A. Feature Importances A-1 Binary Classification and Regression vi

7 A-2 Multiclass Classification vii

8 List of Tables Table I. Wearable systems for PD/LID Assessment Table II. Vision-based systems for PD/LID Assessment Table III. Video durations for each task Table IV. Results of PD assessment benchmark on different models Table V. Results for CPM (MPII + LSP) on each task Table VI. Results of IDPR and CPM on the Extended LSP benchmark Table VII. Summary of relevant features in previous work Table VIII. Joint trajectories used for each task Table IX. List of extracted features Table X. Summary of best correlations for each task and subscore Table XI. Correlations for communication task features vs. UDysRS Table XII. Top features by Spearman correlation for UDysRS communication task Table XIII. Correlations for drinking task features vs. UDysRS Table XIV. Top features by Spearman correlation for UDysRS drinking task Table XV. Correlations for leg agility features vs. UPDRS Table XVI. Top features by Spearman correlation for UPDRS leg agility task Table XVII. Correlations for leg agility features vs. CAPSIT Table XVIII. Top features by Spearman correlation for CAPSIT leg agility task Table XIX. Correlations for toe tapping features vs. UPDRS viii

9 Table XX. Top features by Spearman correlation for UPDRS toe tapping task Table XXI. Floor effects for each task and subscore Table XXII. Summary of papers using machine learning for PD/LID analysis Table XXIII. Hyperparameters for random forest classifier Table XXIV. Hyperparameters for random forest regressor Table XXV. Results for communication task - UDysRS (n = 128) Table XXVI. Results for drinking task - UDysRS (n = 118) Table XXVII. Results for leg agility task - UPDRS (n = 75) Table XXVIII. Results for leg agility task - CAPSIT (n = 110) Table XXIX. Results for toe tapping task - UPDRS (n = 76) Table XXX. Results for UDysRS Part III total score (n = 118) Table XXXI. Results for UPDRS Part III total score (n = 74) Table XXXII. Results for multiclass classification of communication task (n = 77) Table XXXIII. Results for multiclass classification of leg agility task (n = 53) Table XXXIV. Top features for distinguishing onset of dyskinesia Table XXXV. Top features for distinguishing remission of dyskinesia Table XXXVI. Features for distinguishing remission of dyskinesia sorted by correlation to UDysRS ix

10 List of Figures Figure 1. Schematic of common setup of wireless sensors to capture movement of all four limbs Figure 2. Examples of pose estimation results from DeepPose Figure 3. Sample images from the PD assessment benchmark Figure point skeleton annotated in PD assessment benchmark Figure 5. Sample detections from benchmarked methods Figure 6. Detection curves comparing overall PCKh across all methods (left) and comparing CPM (MPII + LSP) performance on different body parts (right) Figure 7. Examples of failed knee detection by CPM Figure 8. Distributions of prediction errors for CPM Figure 9. Supervised and unsupervised learning problems Figure 10. Top two PCA components for multiclass communication (left) and leg agility (right) tasks Figure 11. Performance comparison at different noise levels Figure 12. Joint importance values for regression of communication (UDysRS) task subscores. 76 Figure 13. Tally of features in top 10 from each subscore of communication (UDysRS) task Figure 14. Joint importance values for regression of drinking (UDysRS) task subscores Figure 15. Tally of features in top 10 from each subscore of drinking (UDysRS) task Figure 16. Joint importance values for regression of leg agility (UPDRS) task subscores Figure 17. Joint importance values for regression of leg agility (CAPSIT) task subscores x

11 Figure 18. Tally of features in top 10 from each subscore of leg agility (CAPSIT) task Figure 19. Average joint feature importance values for regression of toe tapping (UPDRS) task subscores Figure 20. Joint importance values for regression of UDysRS Part III total subscores Figure 21. Tally of features in top 10 from each subscore of UDysRS Part III total score Figure 22. Average joint feature importance values for regression of UPDRS Part III total score Figure 23. Joint importance values for multiclass classification of communication task Figure 24. Joint importance values for multiclass classification of leg agility task Figure 25. Schematic of anchors in levodopa infusion protocol xi

12 Acronyms ADL AUC CAPSIT CNN COMT CPM DPM EMG FLIC i.v. IDPR IQR KLT LID LSP m-aims MAO-B MCID MDS MPII PCA PCK PCKh PD PSD PDJ RMS ROC UDysRS UPDRS Activities of Daily Living Area under the Receiver Operating Characteristic Curve Core Assessment Protocol for Surgical Interventional Therapies Convolutional Neural Network Catechol-O-methyl Transferase Convolutional Pose Machines Deformable Parts Model Electromyography Frames Labelled in Cinema Intravenous Image Dependent Pairwise Relations Interquartile Range Kanade-Lucas-Tomasi Levodopa-induced Dyskinesia Leeds Sports Pose Modified Abnormal Involuntary Movement Scale Monoamine Oxidase B Minimally Clinically Important Difference Movement Disorder Society Max Planck Institut Informatik Principal Component Analysis Percent of Correct Keypoints Percent of Correct Keypoints Head Parkinson's Disease Power Spectral Density Percent of Detected Joints Root Mean Square Receiver Operating Characteristic Unified Dyskinesia Rating Scale Unified Parkinson's Disease Rating Scale xii

13 Chapter 1 Introduction 1.1. Motivation Parkinson s disease (PD) is the second most common neurodegenerative disorder [1], affecting more than 4 million people worldwide [2]. The economic burden associated with PD in the US alone exceeds $23 billion per year [3]. As the global population ages, PD will become an even more pressing concern, with prevalence estimated to be over 8 million globally by 2030 [2] and economic costs in the US expected to balloon to $50 billion annually by 2040 [3]. Persons with PD experience a host of motor symptoms including tremor, bradykinesia (slowness of movement), rigidity, and freezing of gait [4]. The standard medication for PD management is levodopa, which is highly effective at neutralizing or reducing motor symptoms [5]. However, levodopa s usefulness is hampered by the appearance of motor complications following prolonged usage. These motor complications are termed levodopa-induced dyskinesias (LID), and usually present as involuntary, irregular motions that flow from one body part to the other, or twisting of body parts into unnatural positions. As a result, drug regimens are often composed of multiple drugs, seeking to maximize antiparkinsonian benefits while minimizing dyskinesia. Currently, there are no FDA-approved drugs for treatment of LID. Due to the heterogeneous nature of PD and LID symptoms between individuals, the selection of medications and dosages is a highly personalized process. To inform treatment selection, people with PD have regular clinic visits and are often asked to keep paper diaries of their symptoms. However, both of these methods may not be fully representative of symptoms. Clinic visits are intermittent and may be subject to white coat adherence, whereby patients closely adhere to their medication schedule in advance of appointments [6]. Clinicians use rating scales to record characteristics of PD motor signs (e.g. anatomical distribution, duration, functional impact). However, these rating scales can be timeconsuming to perform and require specialized training to administer. The experience of the rater can also significantly influence ratings [7]. In addition, past studies have shown that patient compliance with paper diaries is low [8], and interpretations of symptoms can differ significantly 1

14 between patients and physicians [9]. These issues underline the need for more frequent and objective data collection methods to provide more accurate information. An automated assessment system would address issues with existing clinical practices. It would allow patients to assess their symptoms more frequently and provide more useful information to their neurologist regarding drug dosage adjustment. Furthermore, computerized assessments can provide an objective measurement of motor symptoms and therefore would be more consistent than patient diaries or ideally, even neurologist performed clinical ratings. It could also be used as a screening tool to determine if someone is exhibiting signs of PD or LID, and could potentially accelerate clinical trials for new LID interventions. If concurrent validity can be shown between an automated system and current clinical scales for dyskinesia, this system could be used to assess patients much faster. By leveraging telemedicine, this system could also be used to recruit patients in less accessible areas, as distance and the need for frequent clinic visits are often cited as barriers in health research recruitment [10]. While systems based on wearable sensors (e.g. accelerometers, gyroscopes, etc.) have been proposed for automated assessment, it remains to be seen whether they have clinical utility. A potential solution for automated PD/LID assessment is to use computer vision. The goal of computer vision is the extraction of information from images or videos. Applications of computer vision are prevalent in everyday life, such as face detection in cameras, video games with the Microsoft Kinect TM, or automatic bank account number recognition on cheques. Recently, the field of computer vision has developed rapidly, spurred by exponential advances in computer hardware. Vision algorithms can be used to predict human pose and to track motion of the body over time, making them a candidate for assessing involuntary motions in PD. Automated assessments can be performed unobtrusively using cameras, which are relatively inexpensive, readily available, and do not require direct contact with the body. In this thesis, computer vision is proposed for objective assessment of parkinsonism and LID. Movement trajectories of body parts are extracted from video, after which features (e.g. mean velocity, frequency) are computed to describe characteristic movement patterns. The system s goal is to distinguish parkinsonism from LID as well as rating their respective severities on validated clinical scales. 2

15 1.2. Objectives The research question this thesis seeks to answer is: Can automated, unobtrusive, clinical assessments of parkinsonism and LID be accurately performed by leveraging computer vision-based approaches? Towards answering this research question, this thesis will address three specific aims: Aim 1: Investigate the feasibility of state-of-the-art computer vision methods for body movement tracking during clinical PD assessments. State-of-the-art computer vision methods will be applied to videos of clinical assessments in order to estimate pose and body movement. The performance of these methods will be compared to a set of manually annotated joint positions. Expected performance on clinical PD assessment videos has been conservatively estimated based on performance reported in [11]. Hypothesis 1: Computer vision methods will track body position and movements in clinical PD assessment videos with a Percent of Detected Joints (PDJ) > 0.7 at a normalized distance (percent of torso diameter) of 0.2 [11]. Aim 2: Investigate movement features that are most discriminative of parkinsonism and LID and their respective severities. In order to distinguish parkinsonism and LID and estimate their severities, features must be extracted from movement trajectories. Examples of features include joint velocity, acceleration, frequency, or spectral power. Expert neurologist ratings are used as the ground truth for validation. Features will be compared to clinical scores of parkinsonism and LID to determine which are most highly correlated. Hypothesis 2: Features will be discovered with strong correlation with expert ratings, defined as a magnitude of Spearman correlation >

16 Aim 3: Detect the presence or absence of parkinsonism/lid and estimate their respective severities. The prediction tasks are divided into a) classification and b) regression tasks. Expert neurologist ratings are used as the ground truth for validation. a. Predicting the presence or absence of parkinsonism/lid can be framed as three distinct classification tasks parkinsonism vs. normal (binary), LID vs. normal (binary), and parkinsonism vs. LID vs. normal (multi-class). Hypothesis 3a: Binary classification performance will exceed an F-score of 0.8 and area under the receiver operating characteristic curve (AUC) > 0.8. Multi-class classification will have an accuracy of > 7. b. Predicting the severity of parkinsonism/lid consists of two separate regression tasks. Hypothesis 3b: Regression predictions will have a strong correlation with expert ratings, defined as a Pearson correlation >

17 Chapter 2 Literature Review 2.1. Parkinson s Disease (PD) and Levodopa-Induced Dyskinesia (LID) Pathophysiology and Clinical Presentation Parkinson s disease is a neurodegenerative disorder caused by the loss of dopamine producing cells in a brain region called the substantia nigra pars compacta, which is important for motor control. While some PD cases are genetic or caused by exposure to certain chemicals, most cases of PD are idiopathic, that is, the reason for the loss of these cells is unknown [12]. PD is linked to aging, with incidence increasing rapidly over the age of 60, and only 4% of reported cases under the age of 50 [13]. The cardinal features of PD are tremor at rest, rigidity, bradykinesia (slowness of movement) and postural instability [4]. Levodopa, a precursor for dopamine, is the most effective drug for relief of PD motor symptoms. However, after prolonged usage, patients develop involuntary movements called levodopainduced dyskinesias (LID). The amount of time that passes before occurrence of LID varies, although it was reported that 4 of PD patients developed LID within 4-6 years [14]. The precise cause of LID is unknown. One theory is that the intermittent administration of levodopa does not accurately mimic the naturally continuous production of dopamine in the brain [15]. This abnormal stimulation of the brain regions associated with motor control leads to dysfunction. Known risk factors for LID include age of onset of PD and initiation of levodopa therapy, dosage strength, and duration of treatment [16]. LIDs are characterized by their time of occurrence relative to blood plasma concentration of levodopa. The most common type of LID is peak-dose dyskinesia, which occurs when plasma levodopa level is maximal [16]. Peak-dose dyskinesias primarily consist of two types of movements: chorea and dystonia. Choreas are arrhythmic movements that appear to flow from one body part to another, whereas dystonias are involuntary contractions of opposing muscles, causing twisting of the body into abnormal positions [17]. Most PD patients will experience some combination of dyskinesia types, including diphasic dyskinesias and off-period dystonias that occur at different points in the levodopa dosage cycle. 5

18 Treatment Options Levodopa is the standard treatment for PD motor symptoms. It is often combined with other drugs such as carbidopa, monoamine oxidase B (MAO-B) inhibitors or catechol-o-methyl transferase (COMT) inhibitors. These drugs inhibit enzymes that degrade dopamine, thus extending levodopa s action. Unfortunately, the co-administration of these drugs does not delay the onset of LID [18]. Currently, there are no FDA-approved drugs for treatment of LID. However, both amantadine and clozapine have been shown to be efficacious in dyskinesia, although clozapine still poses some safety concerns [19]. Surgical interventions such as deep brain stimulation and pallidotomy significantly reduce LID, but surgery carries inherent risks and some individuals may not be able to tolerate surgery [18]. As LID treatment options are limited, the most effective strategy is to prevent the development of LID. Dopamine agonists can be used instead of levodopa to delay dyskinesia onset but they provide suboptimal relief of PD symptoms. Likewise, levodopa doses can be reduced at the cost of reduced antiparkinsonian effect. Interestingly, a study found that PD patients with LID and adequate antiparkinsonian benefits reported higher quality of life scores than patients without LID experiencing poor or moderate antiparkinsonian benefits [20], illustrating the trade-off between delaying onset of LID and improved quality of life Rating Scales Rating scales are used in PD and dyskinesia assessment to observe disease progression and treatment efficacy. These rating scales must be administered by trained clinicians. A brief description of commonly used scales is provided below. For more detailed information, see [21]. The Unified Parkinson s Disease Rating Scale (UPDRS) is the standard rating scale used in assessment of PD. Introduced in the 1980s, it was revised in 2007 by the Movement Disorder Society (MDS) to address previously identified issues. The modified version is referred to as the MDS-UPDRS and combines historical information from the patient and/or caregiver with the clinician s assessment of PD motor signs [22]. In this thesis, MDS-UPDRS and UPDRS are used interchangeably to refer to the MDS-UPDRS. The MDS-UPDRS contains four parts: 6

19 Part I: Non-motor experiences of daily living patient questionnaire regarding non-motor impacts of PD (e.g. depression, anxiety, sleep problems, fatigue) Part II: Motor experiences of daily living patient questionnaire regarding motor impacts of PD (e.g. tremor, freezing, speech, eating) Part III: Motor examination clinician assessment of PD motor signs (e.g. finger tapping, sit-to-stand, gait) Part IV: Motor complications clinician assessment combined with patient questionnaire to assess impact of dyskinesias and motor fluctuations (e.g. time spent with dyskinesias, functional impact of dyskinesias, painful off-state dystonia) The Rush Dyskinesia Rating Scale (RDRS or Rush) assesses dyskinesia during three tasks walking, drinking from a cup and dressing [23]. Raters mark the severity of dyskinesias on a scale of 0 (no dyskinesia) to 4 (violent dyskinesia), indicate which dyskinesias are visible (e.g. chorea, dystonia, etc.), and which type of dyskinesia is most disabling. The modified Abnormal Involuntary Movement Scale (m-aims) is a clinician assessment of the anatomical distribution and intensity of abnormal movements [24]. Abnormal movements are rated in four categories: (1) facial and oral movements, (2) extremity movements, (3) trunk movements, and (4) global judgments, totalling 10 items. Each item is rated on a scale of 0 (none) to 4 (severe). The Core Assessment Protocol for Surgical Interventional Therapies in Parkinson s Disease (CAPSIT-PD) is a program designed to screen potential candidates for PD surgical interventions and to follow-up after surgery [25]. One of the components of CAPSIT-PD is a dyskinesia rating where the patient performs multiple motor tasks from UPDRS Part III. Seven anatomical regions are rated on a scale of 0 (none) to 4 (extreme), for a maximum score of 28. The Unified Dyskinesia Rating Scale (UDysRS) is a comprehensive rating tool for assessing PD dyskinesias made up of both subjective (Part I, II) and objective (Part III, IV) components [26]. Although it was recently developed, it has already shown excellent sensitivity compared to other dyskinesia rating scales [27]. It contains four parts: 7

20 Part I: Historical Disability of On-Dyskinesia Impact patient questionnaire regarding impact of on-dyskinesia ( on meaning when medication is working) on daily activities (e.g. speech, eating, hygiene). Part II: Historical Disability of Off-Dystonia Impact patient questionnaire regarding impact of off-dystonia ( off meaning when medication is not working). Off-dystonia causes cramps, spasms, and pain that can limit activities. Part III: Objective Impairment clinician assessment of dyskinesia intensity during four tasks: communication, drinking from a cup, dressing, and ambulation. Seven body parts are rated on a scale of 0 (no dyskinesia) to 4 (incapacitating dyskinesia). The body parts are the face, neck, left and right arm/shoulder, left and right leg/hip, and trunk. Part IV: Objective Disability clinician assessment of dyskinesia interference in performing tasks from Part III. Each task is rated on a scale of 0 (no dyskinesia) to 4 (severe interference) Technologies for PD/LID Assessment Rating scales rely on reports from clinicians and patients for scoring. Efforts have been made to use technologies to automatically assess PD and LID. The following is a brief overview additional reviews can be found at [28], [29]. To characterize PD/LID, technologies record signals from participants while replicating tasks from an existing rating scale, performing a predefined protocol based on activities of daily living (ADL), or during their usual daily activities in their home environment. While the selection of ADL varies between studies, ADL can be broadly categorized as relating to bathing, dressing, toileting, transferring (i.e. moving from place to place), and continence [30] Wearable Sensing Wearable systems have been the most popular technology in PD/LID assessment research. Movement kinematics are captured through accelerometers, gyroscopes (tilt) and/or magnetometers (direction). 8

21 Table I provides a summary of notable studies using wearable systems. Figure 1 shows an example of wireless sensor attachment designed to capture motion from all four limbs. Other sensor configurations include placing sensors on the most affected side of the body, or on the chest, back, or head. Figure 1. Schematic of common setup of wireless sensors to capture movement of all four limbs. Images show straps used for sensor attachment and relative size of sensors [31] ( 2011 IEEE). Using a tri-axial accelerometer placed on the shoulder, Manson et al. found that acceleration in the 1-3 Hz range was correlated with clinical dyskinesia scores [32]. Hoff et al. used accelerometers placed on all four limbs and found that the spectral power in specific frequency bands was correlated to m-aims scores in the absence of voluntary motion [33]. Using gyroscopes placed on both forearms, Salarian et al. were able to detect tremor and quantify tremor and bradykinesia severity [34]. Giuffrida et al. demonstrated the Kinesia TM by Great Lakes NeuroTechnologies, a single device (tri-axial accelerometer and gyroscope) placed on one finger capable of predicting UPDRS scores for rest, postural, and kinetic tremor [35]. Further iterations of the Kinesia TM system (Kinesia TM 360) have focused on continuous monitoring using wrist and ankle bands [36]. Patel et al. used accelerometer recordings during tasks from the UPDRS motor assessment to analyze a comprehensive set of movement features and determine which motor tasks were best for predicting tremor, bradykinesia, and dyskinesia severity [31]. 9

22 Using accelerometers placed on the PD patients most affected side, Zwartjes et al. classified a predefined set of motor activities as well as the level of deep brain stimulation patients were receiving [37]. Eskofier et al. analyzed the finger-to-nose and pronation-supination movements to detect bradykinesia [38]. Due to the diversity of motor tasks available for PD assessment, some studies tailored their analysis to single tasks, such as postural stability [39], finger tapping [40], [41], pronation-supination [42], and leg agility [43]. Smartphone accelerometers have also been shown to be sufficiently sensitive to discern individuals with PD from healthy controls [44], [45] and capture movement features significantly correlated to clinical scores [46]. Instead of examining tasks (or a subset of tasks) from or similar to those in clinical rating scales, an alternative approach to PD assessment is to do continuous monitoring. While this approach is more difficult due to the need for activity detection or segmentation, it better mimics the patient s natural environment. Keijsers et al. monitored participants as they performed a predefined protocol of ADL and were able to predict m-aims scores for one-minute time windows [47]. By using accelerometers and electromyography (EMG), Cole et al. were able to detect tremor and dyskinesia during unconstrained monitoring down to a one-second resolution [48]. Hammerla et al. collected data from both a lab and home environment and concluded that lab data was a poor predictor of home data [49]. Table I. Wearable systems for PD/LID Assessment Year Author Subjects Total (PD) Tasks Outcome 2000 Manson et al. [32] 26 (16) sitting, writing, eating, Rush m-aims and Rush scores 2001 Hoff et al. [33] 23 (23) sitting, counting, spelling, Rush m-aims score 2003 Keijsers et al. [47] 13 (13) continuous monitoring (35 ADL protocol) m-aims score 2007 Salarian et al. [34] 20 (10) 17 ADL tremor detection, UPDRS tremor and bradykinesia scores 2009 Giuffrida et al. [35] 60 (60) rest, postural, and kinetic tremors 10 UPDRS tremor scores

23 2009 Patel et al. [31] 12 (12) subset of UPDRS motor examination UPDRS tremor, bradykinesia, and dyskinesia scores 2009 Yokoe et al. [40] 2010 Tsipouras et al. [50] 48 (16) finger tapping UPDRS finger tapping score 10 (7) 9 ADL UPDRS dyskinesia scores 2010 Zwartjes et al. [37] 6 (6) subset of UPDRS motor examination activity classification, UPDRS scores 2011 Palmerini et al. [39] 40 (20) standing classify PD/healthy 2012 Rigas et al. [51] 23 (18) 8-item protocol (including sitting, lying down, walking, picking up object) action/posture recognition and UPDRS tremor score 2013 Mera et al. [52] 15 (15) rest and postural tremor m-aims score 2013 Stamatakis et al. [41] 46 (36) finger tapping UPDRS finger tapping score 2014 Arora et al. [44] 20 (10) gait, standing classify PD/healthy 2014 Cole et al. [48] 12 (8) unconstrained continuous monitoring 2015 Lee et al. [53] 7 (7) subset of UPDRS motor examination tremor and dyskinesia detection, UPDRS tremor and m-aims scores dyskinesia severity (5 point scale) 2015 Pasluosta et al. [54] 139 (139) subset of UPDRS motor examination UPDRS postural stability score 2015 Kostikis et al. [45] 2015 Giuberti et al. [43] 25 (25) rest and postural tremor classify PD/healthy 24 (24) leg agility UPDRS leg agility score 2015 Hammerla et al. [49] 34 (34) unconstrained continuous monitoring classify state (off, on, dyskinesia, asleep) 11

24 2016 Kassavetis et al. [46] 14 (14) subset of UPDRS motor examination UPDRS scores 2016 Piro et al. [42] 15 (15) pronation-supination UPDRS pronationsupination score 2016 Eskofier et al. [38] 10 (10) finger-to-nose, pronationsupination bradykinesia detection While wearable sensing has advantages in being discreet and wireless, it still requires the user to wear the system. Moreover, there is a lack of standardization for the location and number of sensors required. At this time, further validation is required if data derived from wearable technologies are to be used for diagnosis or as clinical endpoints Vision-Based Vision-based assessment of PD/LID has so far been very limited. Table II summarizes published work in this field. Earlier work on vision-based parkinsonian gait analysis used multi-coloured suits to aid segmentation of body parts [55], [56]. Cho et al. used background subtraction to detect the participant frame-by-frame while walking, and then used the resultant silhouettes to classify PD patients and healthy controls [57]. Jobbágy et al. used a combination of IR LED markers and contact sensors to record and segment finger tapping cycles [58]. Another finger tapping study conducted by Khan et al. required participants to hold up their hands on either side of their head so that face detection could be used to approximate their hand position [59]. To observe global dyskinesia, Rao et al. manually landmarked several points on the body and tracked their motion during a communication task using non-rigid image registration [60]. They found that the lack of coordination between limbs was correlated to the UDysRS objective disability score for communication (Part IV). Table II. Vision-based systems for PD/LID Assessment Year Author Subjects Total (PD) Tasks Outcome 2000 Green et al. [55] 18 (10) gait classify PD/healthy 2005 Jobbágy et al. [58] 42 (10) finger tapping classify PD/healthy 12

25 2008 Lee et al. [56] 90 (40) gait classify PD/healthy 2009 Cho et al. [57] 14 (7) gait classify PD/healthy 2010 Roiz et al.* [61] 27 (12) gait classify PD/healthy 2011 Das et al.* [62] 6 (4) subset of UPDRS motor examination UPDRS scores 2013 Rao et al. [60] 35 (35) communication UDysRS Part IV communication score 2014 Dror et al. [63] 13 (8) UPDRS hand movements classify PD/healthy 2014 Khan et al. [59] 19 (13) finger tapping UPDRS finger tapping score 2014 Procházka et al. [64] 2015 Rocha et al. [65] 2016 Růžička et al.* [66] 36 (18) gait classify PD/healthy 9 (4) gait classify PD-stimulator on, PD-stimulator off, healthy 44 (22) finger tapping classify PD/healthy Kinect TM *Optical motion capture Multi-camera systems allow resolving of movement in 3D, providing more accurate motion tracking. Multiple studies have used the Microsoft Kinect TM s built in skeletal tracking to extract gait parameters and identify parkinsonian gait [64], [65]. Dror et al. took advantage of depth sensing for hand segmentation during UPDRS hand movements (i.e. finger tapping, pronation/supination, hand open/close), and extracted movement features to discriminate between PD patients and healthy controls [63]. More sophisticated optical motion capture systems require reflective markers to be attached to points to be tracked, but are much more accurate than markerless solutions. In the context of PD, optical motion capture has been used to characterize gait [61], as well as study various tasks in the UPDRS motor assessment [62]. Růžička et al. found that features derived from motion capture of the finger tapping task were better at distinguishing PD and control subjects than the UPDRS finger tapping score and 13

26 standard instrumental tests of bradykinesia [66]. Unfortunately, most optical motion capture systems are prohibitively expensive and impractical, making them only suitable for research use Other Methods Magnetic motion capture systems track movements using sensors placed on the body that measure the relative strength of a magnetic field from a known transmitter source. While highly accurate, they suffer from the same cost and practicality disadvantages as optical motion capture. Studies using magnetic motion capture have analyzed postural sway and involuntary movements of dyskinetic and non-dyskinetic PD patients while standing [67] [69]. Mann et al. compared PD dyskinesia movement features with Huntington s chorea and found differences in involuntary movement speed and frequency [70]. Lones et al. identified movement patterns characteristic of PD patients and healthy controls during rest tremor and finger tapping [71]. As the focus of this thesis is vision-based assessment of PD/LID, modalities that acquire raw data that cannot be translated for kinematic analysis in videos will not be covered in detail. These include speech analysis [72], radar [73], and digitized spiral drawing tasks [74]. An overview of EMG-based assessment is provided in [28]. Goetz et al. developed a home-based computer module with specialized hardware such as a pegboard, buttons for testing reaction time, and a keyboard for finger tapping [75]. While the system was well received by study participants, it uses custom hardware and has not been tested outside of early PD Computer Vision Methods for Pose Estimation Vision-based pose estimation has been an active field in computer science for several decades. Comprehensive, although somewhat dated (before 2007), reviews of pose estimation and human motion capture can be found in [76] [78]. To represent human motion, the majority of methods employ a model-based approach. Modelbased approaches define a humanoid shape to approximate the kinematic structure, appearance, and shape of the human body. One example of such a model is the deformable parts model (DPM). DPM models the body as a network of parts connected by springs [79]. The spring tension restricts connectivity between parts, thus increasing the likelihood that predicted poses will be valid. Therefore, poses must be optimized such that body parts match the scene and to minimize the spring tension. 14

27 Deep learning methods allow for automatic discovery of data representations useful for discrimination. Recent advances in deep learning have made it the defacto standard for not only pose estimation, but also object recognition, image understanding, and natural language processing [80]. Automatic feature discovery is an advantage of deep learning over conventional machine learning methods, which require domain knowledge in order to design features that will accurately represent underlying patterns in the data. Deep learning is built on neural networks, a biologically inspired machine learning algorithm. Neural networks are composed of layers of neurons that individually perform basic operations. However, when connected together, neural networks are capable of modelling complex structure in data. A specific type of deep learning architecture commonly used for images is the convolutional neural network (CNN). In order to handle the high dimensionality of images and take advantage of their inherent structure, CNNs have particular adaptations that make them much more efficient than regular neural networks. Figure 2. Examples of pose estimation results from DeepPose [11] ( 2014 IEEE). DeepPose was the first application of deep neural networks to human pose estimation, and demonstrated equivalent or better performance than state-of-the-art systems at the time (see Figure 2) [10]. Chen and Yuille used CNNs to learn conditional probabilities for a graphical model representing pose [81]. By incorporating motion information, Pfister et al. smoothed frame-by-frame pose estimates for tracking in video [82]. Tompson et al. introduced a method of refining joint positions by combining the existing body part detector with spatial constraints on joint inter-connectivity [83]. Wei et al. iteratively refined heatmaps of joint positions by using long range interactions between joints [84]. 15

28 As the development of deep learning is recent, many investigations of cross-disciplinary applications are still in their early stages. At this time, the only explorations of using deep learning for assessing PD were by Eskofier et al. and Hammerla et al., both of which were for classification of wearable sensor data [38], [49]. Therefore, excellent opportunities exist for computer vision and deep learning to be applied in PD assessment. 16

29 Chapter 3 PD Assessment Dataset 3.1. Overview This project was conducted in collaboration with the Movement Disorders Centre at Toronto Western Hospital (TWH). The team at TWH, led by neurologists Dr. Tiago Mestre and Dr. Susan Fox, conducted the data collection described in this section. The purpose of their study was to determine the minimally clinically important difference (MCID) in standard parkinsonism and dyskinesia rating scales, that is, the magnitude of change in the scale required for patients to perceive a difference. The results of that study and more detailed protocol are published in [85]. The collected dataset contains videos showing clinical assessments of PD patients, with varying levels of LID, performing a set of motor tasks. These tasks are a part of standard rating scales, and are designed to measure the degree of influence of their motor symptoms on their ability to perform activities of daily living. Examples of tasks include finger tapping, leg stomping, and drinking from a cup. Videos were captured using a consumer grade video camera at 30 fps at a resolution of or Study Population Nine participants completed the study protocol. There were 5 male and 4 female participants with a median age of 64 years. Inclusion criteria 1. Diagnosis of idiopathic PD as per United Kingdom PD Society Brain Bank criteria [86] 2. Age years 3. Stable bothersome levodopa-induced peak-dose dyskinesia for >25% of the day (defined as UPDRS item 4.1, rating 2, Lang-Fahn Activities of Daily Living Dyskinesia Scale 1 [87]) 4. Stable antiparkinsonian medication for one month in advance of study participation 17

30 Exclusion criteria 1. Hoehn & Yahr score of 5 when in off state (confined to wheelchair or bed unless aided) [88] 2. UPDRS score of 3-4 for resting or action tremor when off 3. Cognitive impairment (defined as a score < 24 on the Montreal Cognitive Assessment [89]) 4. Previous surgery for PD 5. Other significant conditions that could interfere with study participation, up to the discretion of the investigator (e.g. medication hypersensitivity, pregnancy, hypotension) 3.3. Study Protocol Participants attended four visits over 7 ± 2 weeks. The first visit was a screening visit, while the following three visits followed a levodopa infusion designed to measure the MCID. An anchorbased approach was applied, where anchors indicated clinically important events as reported by participants. Parkinsonism and dyskinesia rating scales were administered at regular intervals as well as when anchors were reported. Rating scales used were UPDRS Part III, UDysRS Part III, Rush, m-aims, and CAPSIT. Participants were asked to refrain from their usual overnight medication for 12 hours in advance of their visit. Therefore, participants began the visit in the practically defined off state or baseline, where parkinsonism should be maximal and dyskinesia should be minimal. They were given an intravenous (i.v.) infusion of levodopa or placebo for 2 hours. The dosage was determined based on their regular levodopa (or levodopa equivalent) dose. Participants continued to be evaluated until they returned to 5 of their baseline UPDRS Part III scores, or a maximum of 2 hours from the i.v. being turned off. Participants were queried every 15 minutes for awareness of 5 anchors: 1) maximum improvement of parkinsonism, 2) reappearance of parkinsonian symptoms, 3) onset of 18

31 dyskinesia, 4) peak intensity of dyskinesia, and 5) remission of dyskinesia. Assessments were video recorded every 15 minutes until the onset of anti-parkinsonian effect, after which they were performed every 30 minutes. In addition, assessments were performed at all anchor points. Due to the i.v. line, some scales were slightly modified as certain tasks were inconvenient to perform. UPDRS was modified by removal of the rigidity and postural stability tasks while UDysRS and Rush did not include the dressing/undressing task. Two (or three, depending on rating scale) expert raters blind to the treatment (levodopa/placebo) and the time point of the recording (dosage) rated videos afterwards Task Selection Although rating scales were validated based on their clinimetric properties, the individual motor tasks were not. Therefore, the validity of individual tasks was determined based on their Pearson correlation (r) with the total validated assessment score. The selection of tasks for this study was driven by two factors: the validity of the task and the perceived feasibility of assessment by computer vision. For the UDysRS motor assessment (Part III), there were four tasks. For each task, seven parts of the body were rated. The validated scores for Part III were made up of the highest scores recorded for each body part over all tasks. Three of the tasks had ratings and videos available in this dataset. The correlation between the body part score for each task and the highest body part score across all tasks was computed. The average correlations (computed using Fisher z- transformation) by task were: Communication: r = (range: ) Drinking: r = (range: ) Ambulation: r = (range: ) For the UPDRS motor assessment (Part III), the total validated assessment score was the sum of all subscores. The top ranked subscores by correlation are given below: 3.14 Global spontaneity of movement (bradykinesia): r = Leg agility: Right leg r = 0.855, Left leg r = Toe tapping: Right foot r = 0.829, Left foot r = Gait: r =

32 3.12 Postural stability: r = Hand movements: Right hand r = 0.805, Left hand r = Finger tapping: Right hand r = 0.789, Left hand r = Although other rating scales were also used for dyskinesia, the UDysRS was the most recently developed and was recommended by the Movement Disorder Society [21], so it is the sole rating scale used to determine appropriate tasks for dyskinesia assessment. All three tasks for the UDysRS have good correlations with the validated score (r > 0.7). However, the videos for the ambulation task involved the person walking away from the camera and it was often difficult to discern the feet as the person wore a gown. Therefore, the ambulation task was not used, and likewise, the gait rating from the UPDRS was not used. In UPDRS Part III, task 3.14 (global spontaneity of movement) was not attached to a particular task, so it was not used. For postural stability, the standard test is the pull test, where the neurologist quickly pulls the person backwards to check their balance recovery. Due to scene clutter caused by having both the neurologist and the patient in the video, this task would be difficult for vision-based analysis, so it was not selected. In the hand movements task, the patient alternates touching their nose and another object that the neurologist places in front of them. Since videos were recorded head-on, it was difficult to resolve the distance the patient s hand travels, and the neurologist often occludes part of the patient; therefore, the hand movements task was not used. While toe tapping had the benefit of the heel being anchored on the floor, finger tapping characterization was made much more difficult due to free movement of the wrist and was left out of analysis. The list of tasks and their respective ratings were as follows: 1. Communication (UDysRS Part III) 2. Drinking from a cup (UDysRS Part III) 3. Leg agility (UPDRS Part 3.8, CAPSIT) 4. Toe tapping (UPDRS Part 3.7) As the CAPSIT score was also available for the leg agility task, the leg agility task was assessed for both dyskinesia (CAPSIT) and parkinsonism (UPDRS). 20

33 The selected tasks were manually segmented from the videos of complete assessments. Videos containing severe occlusions or camera movement were removed. Information regarding the videos contained in the dataset can be found in Table III. Table III. Video durations for each task Task # of videos Total duration (h:m:s) Average duration (s) Communication 134 1:13: Drinking : Leg agility : Toe tapping :

34 Chapter 4 Benchmarking Deep Learning Pose Estimation Algorithms 4.1. Introduction While the performance of CNNs for human pose estimation on public datasets has been well documented, it is unclear whether this performance will translate to the PD assessment dataset. To address this concern, a benchmarking dataset was generated using representative frames from the PD assessment dataset and pose estimation performance was evaluated by comparing against manual annotations Methods Algorithms Algorithms were selected based on public availability of code and trained models. The algorithms selected were: Graphical model with Image Dependent Pairwise Relations (IDPR) [81] Convolutional Pose Machines (CPM) [84] IDPR was selected as it was one of the earliest CNN based pose estimation algorithms, beating the pioneering DeepPose algorithm, which was the first to use deep learning for pose estimation. CPM was the top performing pose estimation algorithm as of January However, given rapid advancement in the field, it has since been surpassed (although performance gains were marginal). While both algorithms use CNNs, the approach to pose estimation they adopt is different. IDPR uses CNNs to learn conditional probabilities of the presence of parts and their spatial relationships, and then uses this information to build a graphical model representing human pose. In contrast, CPM uses CNNs to directly predict the pose by iteratively refining heatmaps of joint predictions using long range dependencies between joints. In order to use these algorithms to predict pose, they must first be trained on an annotated dataset. While the ideal scenario would be to train the model on a subset of the PD assessment dataset, the annotation of training data is a time intensive process, especially for the large amount of data required for good performance. Instead, many algorithms provide models pre-trained on 22

35 publically available datasets. For the algorithms selected, IDPR provides models trained on the FLIC (Frames Labelled in Cinema) and LSP (Leeds Sports Pose) datasets, while CPM provides models trained on the Extended LSP, FLIC, and MPII (Max Planck Institut Informatik) datasets. CPM also provides an augmented model trained on combined MPII + Extended LSP datasets. As the FLIC dataset only contains upper body annotations, models trained on FLIC were omitted from the evaluation. All annotations are person-centric, that is, the left and right sides are annotated with respect to the person as opposed to the observer. The LSP dataset contains 2000 images taken from Flickr of people playing sports including full body annotations. The same authors later released the Extended LSP dataset, containing images gathered in a similar fashion. The MPII dataset includes 25,000 images extracted from YouTube videos covering 410 human activities Benchmark Figure 3. Sample images from the PD assessment benchmark. Top row from left to right: Communication, drinking from a cup, arms extended Bottom row from left to right: Leg agility, sitting at rest, toe tapping 23

36 The benchmarking dataset was comprised of 1000 images selected from 500 videos in the PD assessment dataset. The videos were evenly distributed among six tasks: communication, drinking from a cup, arms extended, leg agility, toe tapping, and sitting at rest. Two frames were randomly selected in each video. These frames were manually annotated with a 20 point skeleton, modified from the skeleton used by the Microsoft Kinect TM. Landmarked points are shown in Figure 4. Although the benchmark contains a 20 point skeleton, the IDPR and CPM produce a 14 point skeleton. Therefore, joint predictions are benchmarked for the head, neck, shoulders, elbows, wrists, hips, knees, and ankles (shown in blue in Figure 4). Figure point skeleton annotated in PD assessment benchmark. Blue points indicate joints used for evaluation. In all images, the person was upright, seated, and facing the camera. Image widths and heights were between px and px respectively with an average size of px. Images were resized to suit the required dimensions for the algorithm of interest. For IDPR, images were resized to a height of 150 px, while CPM images were resized and padded to px. 24

37 Evaluation Model performance is evaluated using the Percent of Correct Keypoints (PCK, also known as Percent of Detected Joints [PDJ]) and one of its variants PCKh. PCK is the percent of joints that are detected within a specified fraction of the torso diameter (i.e. the distance between the left shoulder and the right hip) [90]. PCKh is a variation of PCK that uses the head length as the normalizing distance instead of the torso diameter, making it more consistent under different viewpoints [91]. PCK and PCKh are reported at fractions 0.2 and 0.5 respectively Results Figure 5. Sample detections from benchmarked methods. Top row: IDPR, bottom row: CPM (MPII + LSP). Figure 5 shows samples of detection results for IDPR and CPM (MPII + LSP) on the PD assessment benchmark. Visually, CPM has much better joint detection than IDPR. Figure 6 provides a graphical comparison of performance between CPM models and IDPR. Detection curves for CPM models all show better performance than IDPR (Figure 6 left). 25

38 Figure 6. Detection curves comparing overall PCKh across all methods (left) and comparing CPM (MPII + LSP) performance on different body parts (right). CPM trained using the MPII + LSP datasets produced the best results on both PCK and PCKh metrics. Figure 6 (right) shows the detection curves for different body parts from CPM (MPII + LSP). From these curves, the head and shoulders were the most well detected joints, while the knees were the most poorly detected. Table IV provides numerical results comparing all models, confirming that CPM models perform much better than IDPR on all body parts. All CPM models exceeded the threshold set in Aim 1 of 0.2 > 0.7. Table IV. Results of PD assessment benchmark on different models 0.2 Head Shoulder Elbow Wrist Hip Knee Ankle Total IDPR 91.2% 91.8% 65.3% 51.6% 31.4% 7.1% % CPM (MPII) 99.9% % 98.2% 95.4% 40.5% 96.7% 88.7% CPM (LSP) 99.9% 99.5% 89.9% 94.7% % 84.3% CPM (MPII+LSP) 99.9% 99.9% 92.8% % 49.5% % 0.5 Head Shoulder Elbow Wrist Hip Knee Ankle Total IDPR % 69.4% 53.8% 33.4% 7.7% 15.2% 53.1% 26

39 CPM (MPII) 99.9% % 99.2% % 90.5% CPM (LSP) 99.8% 99.5% 91.4% 95.6% % 80.4% 85.3% CPM (MPII+LSP) 99.9% 99.9% 93.9% % 56.7% 98.2% 92.1% Table V shows task specific performance with the PCKh metric for the best performing model (CPM trained with MPII + LSP). The communication task had the best average performance, while the arms extended task had the worst performance. Table V. Results for CPM (MPII + LSP) on each task 0.5 Head Shoulder Elbow Wrist Hip Knee Ankle Total Communication 99.4% 99.4% 99.1% 98.8% 97.9% 72.6% 98.2% 95.1% Drinking 99.7% % 99.7% 98.8% 59.6% 95.5% 93.2% Arms extended % 95.8% 95.8% 44.9% 97.9% 85.7% Leg agility % 98.5% 52.1% 99.1% 92.8% Sitting at rest % 51.2% 99.1% 92.2% Toe tapping % 99.7% 98.2% 59.8% 99.1% 93.8% 4.4. Discussion The disparity in performance between IDPR and CPM models can be attributed to multiple factors. The IDPR model was trained on the LSP dataset, while the most comparable CPM model is trained on the Extended LSP dataset, which contains five times more images. From this PD dataset benchmark alone, it is unclear how much of the performance difference is from a larger training set and how much is from CPM being a more sophisticated and recent algorithm. Instead, it is possible to look at the performance of IDPR and CPM on the Extended LSP benchmark [92]. Table VI shows that even when trained on the same dataset, CPM handily outperforms IDPR on detection of all body parts. Based on this benchmark alone, it is sensible to expect CPM to be better than IDPR on the PD dataset. However, the goal is to determine if 27

40 performance on the PD dataset will be adequate, not which method is the best. In Aim 1, the threshold for acceptability for a pose estimation method was a 0.2 of 7, which IDPR does not achieve. Table VI. Results of IDPR and CPM on the Extended LSP benchmark (reproduced from [92]) 0.2 Head Shoulder Elbow Wrist Hip Knee Ankle Total IDPR 91.8% 78.2% 71.8% 65.5% 73.3% 70.2% 63.4% 73.4% CPM 97.8% 92.5% 88.1% 85.2% 92.2% 91.4% 88.7% 90.7% By comparing the performance of CPM with different trained models, insight can be gained into which datasets generalize better to the PD dataset. While the difference in dataset size partially explains why MPII performs better than Extended LSP, the nature of the datasets is different. LSP focuses on sports related poses, and therefore has very few images of people sitting. In contrast, the MPII covers a broad range of activities, many of which include sitting. The PD dataset benchmark is comprised entirely of participants who are seated, so this difference in training sets is likely a large driver for decreased performance of CPM (LSP) on the lower limbs. Both methods experience deterioration in performance towards more distal body parts (i.e. wrist, ankle) on the LSP benchmark. As distal body parts have more deformability and errors in detection can propagate down the kinematic chain, this decrease in detection rate is expected. However, this behaviour was not consistently observed in the PD dataset benchmark. Part of the reason for this is based on different algorithm architectures for IDPR and CPM. While IDPR links body parts together explicitly using a graphical model, CPM uses long range interactions between body parts to predict pose. This means that CPM takes advantage of easily detectable parts, such as the head and shoulders, to infer the position of other joints, which allows for good accuracy for distal parts even if more proximal parts are not detected correctly. With IDPR, the trend of decreasing performance was observed from shoulder-elbow-wrist. Knee detection was poor for both IDPR and CPM. The knee was difficult to detect due to insufficient texture, which was exacerbated when the participant s leg was straight and there was a dark pants colour. Errors 28

41 were also made when the gown the participant was wearing covered the knees. Examples of failure cases for knee detection by CPM are shown in Figure 7. Figure 7. Examples of failed knee detection by CPM. Task specific performance was consistent across all body parts with two exceptions. The elbow detection in the arms extended task was noticeably worse than for other tasks. During the arms extended task, the participant s hands pointed towards the camera, making the shoulder, elbow, and hand close together in the video. The hand could occlude the elbow, making it difficult to detect. Another anomaly was that knee detection in the communication task was more accurate than for other tasks. One possibility was that when participants were performing the communication task, they were not actively suppressing involuntary motions, so leg dyskinesia sometimes caused their legs to move into positions that were easier to detect. Despite the drinking task involving occlusion of body parts when reaching for the cup, there was no decrease in detection rate compared to other tasks. This highlights one of the advantages of CNN-based pose estimation; they are able to infer the position of joints that may not be visible. This is in contrast to more basic image registration approaches, which are unable to handle temporary occlusion [93]. In order to examine if there were any particular directional biases in pose estimates, the errors between the predictions from CPM and the ground truth are plotted in Figure 8. Distances are normalized to head length. One outlying sample was removed as the ground truth annotation was incorrect. Visually, the distribution of errors for the head, neck, shoulders, ankles, and wrists do not appear to have any major skew. In contrast, the knees have an elongated distribution in the vertical direction, showing many errors downwards. As seen in Figure 7, failed knee detection 29