Reinforcement Learning in RAT, Approaches

Transcription

1 Machine Learning in Robot Assisted Therapy (RAT) Applications of Supervised, Unsupervised and Reinforcement Learning in RAT, Approaches Alexander Kohles Lennart Aigner SS 2018

2 Main Question How to improve the efficiency of diagnosis and therapy of children with Autistic Spectrum Disorder using machine learning techniques? 2

3 Agenda 1. Diagnosis 1. Machine Learning with a Diagnostic Tool 2. Machine Learning with Brain Images 2. Robot Assisted Therapy / Robot Enhanced Therapy 1. Robot Assisted Therapy 2. Robot Enhanced Therapy 3. Adaptive Robot Assisted Therapy 4. Reinforcement Learning in RAT 3. Robot Assisted Therapy Using Diagnosis Data 3

4 1. Diagnosis Use of machine learning for ASD classification Prof. Dr.-Ing. habil. Alois Knoll 4

5 Diagnosis - State of the Art Diagnosis for ASD with a diagnostic tool Autism Diagnostic Observation Schedule - Generic (ADOS-G) Standardized assessment of Social interaction Communication Play Imaginative use Four 30-minute modules [Lord (2000) P. 205 et seq.] 5

6 Diagnosis - State of the Art [Lord (2000) P. 207] 6

7 Diagnosis - State of the Art Free play Snack Response to name Response to joint attention Birthday party [Lord (2000) P. 207] 7

8 Diagnosis - State of the Art Reference: Lord (2000), P

9 Diagnosis - State of the Art Shared enjoyment Use of other s body to communicate Pointing Showing Frequency of vocalization directed to others Spontaneous imitation of joint attention Overactivity Negative behavior Anxiety Reference: Lord (2000), P

10 Diagnosis - Approach Supervised and Unsupervised Learning Supports Making a Diagnosis for ASD 10

11 Diagnostic Tool Binary classification based on data from a typical diagnostic tool Supervised learning for prediction of future cases Data: Test / test evaluation Label: Diagnosis Classification accuracies up to 95% Helps saving time during diagnosis Supports the therapist in difficult cases Feature reduction and ranking of input data Insight for the therapist into the most important features [Thabtah (2018)] 11

12 Brain Image Classification Investigation of brain images Resting state functional magnetic resonance images of the brain s neural patterns provided by the Autism Imaging Data Exchange I (ABIDE I) dataset 1112 datasets, including 539 from individuals with ASD and 573 from typical controls Classification Task Combination of supervised and unsupervised learning (Deep Learning) Dividing the brain images into 200 regions Building a correlation matrix each with a correlation value (-1 to 1) Resulting numbers of features: [Heinsfeld (2018), ABIDE Web-Page] 12

13 Brain Image Classification Deep Neural Network (DNN) for binary classification Reference: Heinsfeld (2018) Highly correlated areas for ASD subjects 13

14 Brain Image Classification Success rates up to 97% (using small datasets) Mean classification accuracy of 70% Correlations between patterns of brain activations and the classification result Insight for the therapist [Heinsfeld (2018)] 14

15 Machine Learning for Diagnosis - Results Success rates of up to 95% for classification with diagnostic tool Success rates of up to 97% for classification with brain images Useful insight for the therapist through feature reduction and correlations Generation of a faster diagnosis tool without the workload of a human for evaluation 15

16 Machine Learning for Diagnosis - Limitations Diagnosis with brain images Lack of more heterogeneous datasets to represent clinical heterogeneity Noisy data due to different measurement devices and equipment Anxiety during imaging process Diagnosis with diagnostic tool Decreasing classification accuracy (about 70%) when using larger datasets Bad in generalization No availability for a big shared database for a diagnostic tool Binary classification not sufficient for a complex disease like autism Different degrees possible Ignoring similar diseases like ADHD or Asperger Syndrome [Heinsfeld (2018), Thabtah (2018)] 16

17 Machine Learning for Diagnosis - Future Work Improvement of the machine learning techniques for larger databases Developing a non-binary classification method for ASD Agreeing / developing a standard diagnostic tool used worldwide Introducing a shared database (like ABIDE I) for the standard diagnostic tool 17

18 Machine Learning for Diagnosis - Recap Use of machine learning techniques for a standard diagnostic tool Use of machine learning techniques for brain images Promising results for small databases Knowledge gain by feature reduction and ranking of the input 18

19 2. Robot Assisted Therapy State of the art of robot assisted and enhanced therapy and an approach to improve it Prof. Dr.-Ing. habil. Alois Knoll 19

20 Reasons for Robot Assisted Therapy Sometimes difficulties in the interaction between ASD patient and humans Detection of a better social understanding during therapies with animals Use of robots: Retaining the simple communication methods of animals Overcoming the disadvantages of animals Stimuli reduction to the most important ones Repetition of stimuli without tiredness Better response to technological devices than to stimuli from humans Acceptance for supervised RAT very high, other than a complete replacement [Thill (2012), Richardson (2018), Scassellati (2012)] 20

21 Robot Assisted Therapy - Wizard of Oz Wizard of Oz - Setup Shortcomings: Autonomy, need of second professional Good performance regarding adaptivity and personalization 21

22 Robot Assisted Therapy - Approach Reinforcement Learning improves capability of adaptation and personalization of Robot Assisted Therapy 22

23 From RAT to Robot Enhanced Therapy (RET) Evolution of Robot Assisted Therapy Robot should be able to infer and interpret child s intention (Utopia: State of mind) Utilization of behavior based architectures Following behaviors had been targeted: Turn taking Imitation Joint attention Performance measure in the task: Primary variables: Primary: e.g. task solving accuracy Secondary variables: Social engagement ( e.g. eye- contact frequency ), emotional level, behavioural level, cognitive level [Esteban et. al. 2017] 23

24 Supervised Autonomy Sensing and Interpretation Child behavior classification Social cognitive controller 24

25 Supervised Autonomy Sensing and interpretation (using external cameras and kinects) Gaze estimation Frequency and time of fixation to surrounding Human action recognition Track joint positions Normalization SVM Action recognition Facial expression recognition Local Binary Patterns on Three Orthogonal Planes (LBP-TOP) SVM Identity and facial expression Object tracking Engagement of child with toys Audio processing Microsoft Kinect Labeling of voice signal automatically (measure of engagement level) 25

26 Supervised Autonomy Child Behaviour Classification Appraising current behavior of child Primitive extraction: Data mapped onto relevant behaviors Annotated data for child-robot interaction Behavior classification: Classifiers generate real-time annotations Validation by therapists annotation Output fed into cognitive controller of robot [P. Esteban et al 2016] 26

27 Supervised Autonomy Social Cognitive Controller Input: Classified sensor data Autonomous Controller General: Follow high level scripts for therapy Socially reactive subsystem Low level processes (determined) Actions creating Deliberative subsystem Child behavior + Therapy script + Therapist input Next robot action Decides autonomous to change if needed (e.g. reengage child) Self-monitoring system Internal alarm: Cannot act due to technical or ethical reasons 27

28 RET - Summary Higher level of autonomy achieved Knowledge of the therapist used to teach robot appropriate actions Suggestion / correction mechanism Learning from demonstration Learn from therapist to reduce his workload Full theory of mind not realistic Identification of well-defined indicators of the child s level of engagement Prof. Dr.-Ing. habil. Alois Knoll 28

29 Adaptive Robot Assisted Therapy Modifying a learned policy to cope with different users Co-adaptation in man-machine interaction systems Use of interactive reinforcement learning methods to facilitate the policy adaptation Robot keeps track of user s score during therapy session Score can be defined by the therapist Idea: Childs engagement, child doing what the robot expected to do ( Wave at me -> child waves) Modeling robots behavior: Markov Decision Process Current task state Agent needs to learn optimal policy [Konstantinos Tsiakas and Maria Dagioglou. Adaptive Robot Assisted Therapy Using Interactive Reinforcement Learning. In ICSR, 2016] Prof. Dr.-Ing. habil. Alois Knoll 29

30 Adaptive Robot Assisted Therapy Use of Interactive Reinforcement Learning (considers human in agents learning process) Learning from Feedback - Personalization factor - Provided by user implicitly through sensing system (face, eye-tracking, engagement level) Learning from Guidance - Safety factor - Therapist altering the policy - Important Use Case: Uncertainty [Konstantinos Tsiakas and Maria Dagioglou. Adaptive Robot Assisted Therapy Using Interactive Reinforcement Learning. In ICSR, 2016] Prof. Dr.-Ing. habil. Alois Knoll 30

31 Reinforcement Learning in RET Combine adaptive approach with RET Use of sensing of RET to get state of environment / setting Use of behavioral model as base Goal: Adapt the already existing user model inside the robot to the specific child Proposed reward function: Child s performance Level of engagement Task performance (e.g. imitation tasks) Use metric of successful session Therapists feedback and guidance during session Child does what robot expects it to do Final reward given by therapist due to his personal meaning Therapist remains the final instance Prof. Dr.-Ing. habil. Alois Knoll 31

32 Reinforcement Learning in RET Limitations Rely on exploration which could end with negative impact actions Still relies on clinical setting Labeling of quality of an action will be always difficult Physical limitations of the robot (Child destroying Robot) Future Work: More reliable data Simulation Sensing without clinical setting Improve therapy atmosphere 32

33 3. Robot Assisted Therapy Supported by Diagnosis Data Input State of the art of robot assisted and enhanced therapy using diagnosis data. A new approach Prof. Dr.-Ing. habil. Alois Knoll 33

34 Use of Diagnosis Data for Improvement of RAT 34

35 Use of Diagnosis Data for Improvement of RAT Integration of data into the robot model: Specific scores during diagnosis with extreme values Include in behavioral model / social controller of robot Advantages Adjusting the robot s behavior to the child better personalization Better understanding of relations between robot and child s reaction Faster adaptation Avoid negative experiences of the child related to the robot Limitations: Training process needed 35

36 Bibliography Thabtah, F. (2018). Machine learning in autistic spectrum disorder behavioral research: A review and ways forward. Informatics for Health and Social Care, 00(00), Heinsfeld, A. S., Franco, A. R., Craddock, R. C., Buchweitz, A., & Meneguzzi, F. (2018). Identification of autism spectrum disorder using deep learning and the ABIDE dataset. NeuroImage: Clinical, 17(August 2017), Thill, S., Pop, C. A., Belpaeme, T., Ziemke, T., & Vanderborght, B. (2012). Robot-Assisted Therapy for Autism Spectrum Disorders with (Partially) Autonomous Control: Challenges and Outlook. Paladyn, Journal of Behavioral Robotics, 3(4). Lord, C., Risi, S., Lambrecht, L., Cook, E. H. J., Leventhal, B. L., DiLavore, P. C., Rutter, M. (2000). The Autism Diagnostic Schedule Generic: A standard measures of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), Scassellati, B., Henny Admoni, & Matarić, M. (2012). Robots for Use in Autism Research. Annual Review of Biomedical Engineering, 14(1), Alpaydin, E. (2010). Introduction to Machine Learning Second Edition. The Adaptive Computation and Machine Learning. Konstantinos Tsiakas, Maria Dagioglou, Vangelis Karkaletsis, Fillia Makedon (2016): Adaptive Robot Assisted Therapy Using Interactive Reinforcement Learning Pablo Gomez Esteban, Hoang-Long Cao, Albert De Beir, Greet Van de Perre, Dirk Lefeber, and Bram Vanderborght. A multilayer reactive system for robots interacting with children with autism ABIDE Web Page: 36

37 37

38 BACKUP INFO 38

39 Diagnosis - Explanations Machine Learning - Used for tasks on datasets with no existing function between input and output - Extraction of the Algorithm or a useful approximation by the Computer - Helps to make predictions for new datasets Supervised Learning - Using labeled Data Both input and output values are given - Used for classification problems and prediction Unsupervised Learning - Only input data is provided - Used to find regularities or structures within the input data Clustering - More objectivity Reference: Alpaydin (2010), pp. 1-4 Reference: Alpaydin (2010), pp Reference: Alpaydin (2010), pp

40 Reinforcement Learning Basics General setting Policy inside RL Agent that maps state of the robot to the next action, trying to optimize final reward reward depends on states and its current value by a reward function reward function crucial part Optimal Policy Learning Q-Learning Prof. Dr.-Ing. habil. Alois Knoll 40

41 Core Ideas How to improve the efficiency of diagnosis and therapy of children with Autistic Spectrum Disorder? Support Therapist during diagnosis with machine learning techniques Analyzing brain images Analyzing a dataset from a typical diagnostic tool Extracting features to support understanding of ASD Increase performance of Robot Assisted Therapy using reinforcement learning Improving individual s abilities and reducing their symptoms through early (cognitive-) behavioral intervention programs: Improve language and social skill while decreasing stereotypical behaviors Individuals tend to be more responsive to feedback coming from interaction with technology Wizard of OZ RAT RET 41

42 Agenda 1. Diagnosis a. State of the Art: The Diagnostic Tool b. Hypothesis: Supervised and Unsupervised Learning helps making the diagnosis easier c. Related Work d. Proof of the Hypothesis e. Limitations f. Future Work 2. Robot assisted therapy/ Robot Enhanced Therapy a. State of the Art: Use of Robots in Therapy, Wizard of Oz, Robot Enhanced Therapy b. Hypothesis: Use of reinforcement learning will increase autonomy and personalization c. Related Work d. Reinforcement Learning Environment i. General Design ii. Reward Function e. Discussion (aka The good the bad the future) 3. Robot assisted therapy using diagnosis data a. hypothesis: Combination of both will increase accuracy of diagnosis and personalization of robot enhanced therapy i. General Structure ii. Blabla b. Proof: [paper Esteban et. al is using some, ] 42