Team-Project Geometry and Probability for motion and action

Size: px

Start display at page:

Download "Team-Project Geometry and Probability for motion and action"

Georgiana Ramsey
5 years ago
Views:

1 E-Motion Team-Project Geometry and Probability for motion and action INRIA Grenoble Rhône-Alpes & Laboratory of Informatics of Grenoble (LIG) Scientific leader : Christian LAUGIER (DR1 INRIA) christian.laugier@inrialpes.fr Christian LAUGIER e-motion project-team 1

Overall challenge Context & Scientific challenge Human-Centered Robotics ITS for improving safety & comfort & efficiency Personal Assistant & House Keeping & Rehabilitation Main

Automotive industry, Robots, Health sector, Services Challenging research topics => Dynamic world, Robust perception, Safety, Human Aware Motion, Complex Human-Robot interactions

2 Overall challenge Context & Scientific challenge Human-Centered Robotics ITS for improving safety & comfort & efficiency Personal Assistant & House Keeping & Rehabilitation Main Motivations Important socio-economic perspectives => Transport, Aging society, Medical care & Rehabilitation, Human assistance, Intelligent home Increasing interest of industry => Automotive industry, Robots, Health sector, Services Challenging research topics => Dynamic world, Robust perception, Safety, Human Aware Motion, Complex Human-Robot interactions Robotics state-of-the-art & Progress in ICT Technologies (computers, sensors, micronano technologies, energy ) make this challenge potentially reachable Christian LAUGIER e-motion project-team 2

Robotics Technologies Current Limitations Current Autonomous robots are able to exhibit quite impressive skills. BUT they are NOT adapted to human environments and they are often UNSAFE!

3 Robotics Technologies Current Limitations Current Autonomous robots are able to exhibit quite impressive skills. BUT they are NOT adapted to human environments and they are often UNSAFE! => DARPA Grand Challenge 2004 Significant step towards Motion Autonomy But still some Uncontrolled Behaviors!!!! => URBAN Challenge 2007 A large step towards urban road environments But still some accidents, even at low speed!!! Some technologies are almost ready for use in some restricted and/or protected public areas.. BUT Open environments are still beyond the state of the art Safety is still not guaranteed Too many costly sensors are still required

New challenges to be addressed Human environment

e.g. Traffic scene Dynamicity & Uncertainty

Driver monitoring Human beings are unbeatable in

Technology is better for simple but fast control

4 New challenges to be addressed Human environment understanding e.g. Traffic scene understanding Dynamicity & Uncertainty Interpreting ambiguities Prediction of future states => Avoiding future collisions!!! Share control & Safe Interaction with human Driver monitoring Human beings are unbeatable in taking decisions in complex situations Technology is better for simple but fast control decisions (ABS, ESP ) Patient Wheelchair interaction Human Robot interaction issue has to be addressed

5 Robotics Experimental Platform Commercialized by Robosoft Parkview Cycab & Simulator Commercialized by Bluebotics Equipped Toyota Lexus Koala Autonomous Wheelchairs Industrial Experimental Vehicles Christian LAUGIER e-motion project-team 5

Main Topics & Achievements (2005 (2005-09) 09) 11 PhD, 2 books, 20 journal

(coop. Probayes & Toyota) Robust Perception (BOF) & DATMO (Coop.

Prima) Perception & Situation awareness (5 theses) Learning & World change

ETH Zurich) Prediction & Risk assessment (2 theses) Partial Motion

symmetry (appli. to calibration) Fusion of IMU & Vision (Coop.

on going theses) Risk-based navigation Autonomous navigation & Safety (3

6 Main Topics & Achievements (2005 ( ) 09) 11 PhD, 2 books, 20 journal papers, 3 patents & technological transfers Probabilistic Risk Assessment (coop. Probayes & Toyota) Robust Perception (BOF) & DATMO (Coop. Probayes & Toyota) Vision based Detection & TTC (Coop. Prima) Perception & Situation awareness (5 theses) Learning & World change prediction (coop. ETH Zurich) Prediction & Risk assessment (2 theses) Partial Motion Planning & Inevitable Collision States Observation Prediction Continuous symmetry (appli. to calibration) Fusion of IMU & Vision (Coop. ETH Zurich & Bluebotics) Parameters estimation from noisy sensor data (2 on going theses) Risk-based navigation Autonomous navigation & Safety (3 theses) Efficient 3D Multi-resolution Mapping & Localization using Tensor maps (coop. Perception, IBEO) Dense Mapping & Localisation (1 thesis) Christian LAUGIER e-motion project-team 6

Main Topics & Achievements (Theme 3, 2005 (Theme

(Robotics & Neurosciences), 1 start-up (Probayes)

& Attention focusing [Koike 06] Living systems

Brain controlled wheelchair [Rebsamen 09] (coop.

7 Main Topics & Achievements (Theme 3, 2005 (Theme 3, ) 09) 6 PhD, 1 Book, 9 journal papers (Robotics & Neurosciences), 1 start-up (Probayes) Robots Prey & Predator scenario Action selection & Attention focusing [Koike 06] Living systems Bayesian learning [LeHy 07] [Dangauthier 08] Brain controlled wheelchair [Rebsamen 09] (coop. NUS Singapore) Bayesian models of Superior Colliculus [Colas et al. 09] (coop. LPPA) Human Perception of Shape from Motion [Colas 06] (coop. LPPA) Sensory-motor systems & Handwriting [E. Gillet PhD Thesis] (coop. LPN) Christian LAUGIER e-motion project-team 7

8 Key perception concepts for safe navigation: Situation Awareness & Risk Assessment Probabilistic Risk Assessment Bayesian Perception Equipped Toyota Lexus Stereo camera Ibeo Lux IMU + GPS + Odometry ADAS & Autonomous Driving => Cooperation Probayes, Toyota, Renault Human-Centered Navigation => INRIA PAL & ICT-Asia PAMM Christian LAUGIER e-motion project-team 8

Bayesian Perception Processing Uncertainty & Dynamicity Bayesian Occupation Filter paradigm (BOF) Patented by INRIA & Probayes,

Filtering Estimates at each time step the Occupation & Velocity probabilities of each cell in a space-velocity grid Uses Probabilistic Sensor

05] Application to Detection & Tracking (coop Toyota & Denso) Occupancy grid Unobservable space Concealed space ( shadow of the obstacle)

9 Bayesian Perception Processing Uncertainty & Dynamicity Bayesian Occupation Filter paradigm (BOF) Patented by INRIA & Probayes, Commercialized by Probayes BOF Continuous Dynamic environment modelling using one or several sensors Grid approach based on Bayesian Filtering Estimates at each time step the Occupation & Velocity probabilities of each cell in a space-velocity grid Uses Probabilistic Sensor & Dynamic models => More robust to Sensing errors & Temporary occultation => Designed for Sensor Fusion & Parallel processing [Coué & al IJRR 05] Application to Detection & Tracking (coop Toyota & Denso) Occupancy grid Unobservable space Concealed space ( shadow of the obstacle) Prediction Free space Occupied space Estimation Sensed moving obstacle P( [O c =occ] z c) c = [x, y, 0, 0] and z=(5, 2, 0, 0) Christian LAUGIER e-motion project-team

10 Driving Experimentations INRIA Lexus Platform Inertial sensor / GPS Xsens MTi-G Dell computer + GPU + SSD memory Stereo camera TYZX Toyota Lexus LS600h 2 Lidars IBEO Lux GPS track example (Using Open Street Map)

11 Sensor Fusion experiment: Stereo + 2 Lidars Movie [Perrollaz et al 10] [Paromtchik et al 10] Front view from left camera Fusion result using BOF OG from left Lidar OG from right Lidar OG from Stereo

Conservative Collision Anticipation using the BOF Tracking + Conservative hypotheses Autonomous Vehicle Parked Vehicle (occultation) Thanks to the prediction capability

12 Conservative Collision Anticipation using the BOF Tracking + Conservative hypotheses Autonomous Vehicle Parked Vehicle (occultation) Thanks to the prediction capability of the BOF technology, the Autonomous Vehicle anticipates the behavior of the pedestrian and brakes (even if the pedestrian is temporarily hidden by the parked vehicle)

Collision Risk Assessment Problem statement Behavior Prediction + Probabilistic Risk Assessment Previous observations False alarm! Conservative hypotheses TTC-based crash warning is not sufficient!

13 Collision Risk Assessment Problem statement Behavior Prediction + Probabilistic Risk Assessment Previous observations False alarm! Conservative hypotheses TTC-based crash warning is not sufficient! Consistent Prediction & Risk Assessment requires to reason about : History of obstacles Positions & Velocities (perception or communications) Obstacles expected Behaviors e.g. turning, overtaking, crossing... Road geometry e.g. lanes, curves, intersections using GIS

14 Collision Risk Assessment Functional Architecture Patent INRIA & Toyota 2009 [Tay 09] [Laugier et al 11] Estimate the probability of the feasible driving behaviors Probabilistic representation of a possible evolution of a car motion for a given behavior Probabilistic Collision Risk: Calculated for a few seconds ahead from the probability distributions over Behaviors Recognition & Realization

Learn & Predict Learn => GHMM & Topological maps (SON) Predict =>

15 Motion Prediction: Learn & Predict paradigm Euron PhD Thesis Award 07 (Dizan Vasquez) Observe & Learn typical motions Continuously Learn & Predict Learn => GHMM & Topological maps (SON) Predict => Exact inference, linear complexity Experiments using Leeds parking data

er & Mekhnacha 11] Behavior Prediction e.g.

16 Probabilistic Collision Risk Assessment PhD Thesis Tay Meng Keat + Patent Toyota & Inria & Probayes (2010) Behaviors : Hierarchical HMM (learned) [[Tay & Laugier & Mekhnacha 11] Behavior Prediction e.g. Overtaking => Lane change, Accelerate Motion Execution & Prediction : Gaussian Process GP: Gaussian distribution over functions Prediction: Probability distribution (GP) using mapped past n position observation Christian LAUGIER Keynote FSR 09, Boston

Ov ertaking TurningLeft TurningRight ContinuingStraightAhead Behaviour Probability + Overtaking TurningLeft TurningRight

(Gaussian Process) Experimental validation: Toyota Simulator + Driving device Ego vehicle High-level Behavior prediction

Prediction 0 Behavior models 0,6 Behavior belief table 0,6 Risk Assessment (GP) 0,5 0,4 0,3 0,2 0,1 0 Behavior belief

17 Ov ertaking TurningLeft TurningRight ContinuingStraightAhead Behaviour Probability + Overtaking TurningLeft TurningRight ContinuingStraightAhead Behaviour Probability Collision Risk Assessment Simulation results Ego vehicle Risk estimation (Gaussian Process) Experimental validation: Toyota Simulator + Driving device Ego vehicle High-level Behavior prediction for other vehicles (Observations + HMM) An other vehicle Behavior Prediction (HMM) Observations + 0,5 0,4 0,3 0,2 0,1 Prediction 0 Behavior models 0,6 Behavior belief table 0,6 Risk Assessment (GP) 0,5 0,4 0,3 0,2 0,1 0 Behavior belief table for each vehicle in the scene Evaluation Road geometry (GIS) + Ego vehicle trajectory to evaluate Collision probability for ego vehicle

Odometry Behaviors prediction on a highway (Real time)

18 Collision Risk Assessment Experimental results (Real data) Equipped Toyota Lexus Stereo camera Ibeo Lux IMU + GPS + Odometry Behaviors prediction on a highway (Real time) Cooperation Toyota & Probayes Performance summary (statistics)

Scenario Maneuvers prediction at roads intersections Cooperation Stanford & Renault A vehicle is approaching, then crossing an intersection Available information => perception, previous mapping,

19 Scenario Maneuvers prediction at roads intersections Cooperation Stanford & Renault A vehicle is approaching, then crossing an intersection Available information => perception, previous mapping, communication... Digital map of the road network State of the vehicle: position, orientation, turn signal Associated uncertainty Objective [Lefevre & Laugier & Guzman IV 11] At any t, estimate the manoeuvre intention of the driver of the approaching vehicle (e.g. turn left), using states information and extracted information from the digital map

Maneuvers Prediction at Road Intersections Cooperation Stanford & Renault Digital map obtained using Google Map, an annotated using the RNDF format Typical paths are obtained with a 3D laser

20 Maneuvers Prediction at Road Intersections Cooperation Stanford & Renault Digital map obtained using Google Map, an annotated using the RNDF format Typical paths are obtained with a 3D laser (velodyne), by observing real traffic Intersection 1 Intersection 2 Stanford s Junior Vehicle (parked) 40 recorded trajectories have been manually annotated 2 datasets have been constructed with these trajectories, by automatically annotating the turn signal 40 trajectories with consistent turn signal 40 trajectories with inconsistent turn signal

21 Experimental evaluation : Qualitative Results Consistent turn signal Inconsistent turn signal

22 Experimental evaluation : Quantitative Results Definitions m A, m B = most probable manoeuvre and second most probable manoeuvre Undecidable prediction: P(m A ) - P(m B ) 0.2 Incorrect prediction: P(m A ) - P(m B ) > 0.2 and m A is incorrect Correct prediction: P(m A ) - P(m B ) > 0.2 and m A is correct Results on 2 datasets (40 trajectories each) Consistent turn signal Inconsistent turn signal Entrance Exit

23 e-motion contributions on Mobility Assistance Anne Spalanzani Arturo Escobedo Jorge Rios Martinez Christian Laugier INRIA Rhône-Alpes

24 Navigation of a wheelchair taking into account the context of use : main challenges Study of the needs : who might benefit from an Autonomous Wheelchair? The wheelchair is a robot : autonomous navigation Uncertain and incomplete knowledge of the environment Ability to predict the behavior of the obstacles (which can be humans) The wheelchair transports a person Person/wheelchair communication Integration of social conventions in the navigation decision Autonomous/Semi-autonomous navigation Validation of the proposed system

Scientific challenges The static environment is unknown Construction of maps of the environment Mobile obstacles are not known, but they follow typical patterns Detection & Tracking + Prediction

25 Scientific challenges The static environment is unknown Construction of maps of the environment Mobile obstacles are not known, but they follow typical patterns Detection & Tracking + Prediction on-line Learning of typical patterns Deal with dynamic and uncertain environments Navigation decisions based on a risk criteria (Risk-RRT Fulgenzi 08) Social conventions with proxemics constraints (Personal Space, Interaction) (Rios 2011) Fulgenzi C., Tay C., Spalanzani A., Laugier C. Probabilistic navigation in dynamic environment using Rapidly-exploring Random Trees and Gaussian Processes, IEEE/RSJ 2008 International Conference on Intelligent RObots and Systems, 2008 Rios-Martinez, J., Spalanzani, A., Laugier, C.: Probabilistic autonomous navigation using risk-rrt approach and models of human interaction. In: Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (2011)

26 Risk-RRT (C. Fulgenzi) 26

27 Human aware Navigation (J. Rios Martinez) Personal Space Space of interaction Navigation among humans based on risk and comfort P conf (q) = P coll (q) P pers P inter Not taking into account interactions Taking into account interactions

28 28 Mobility Assistance (A. Escobedo) Navigation system adapted to the person (elder people, disabled, poly disabled ) Autonomy-semi autonomy Interacting with a wheelchair From following to accompanying

Bayesian Perception & Decision for Intelligent Mobility

Bayesian Perception & Decision for Intelligent Mobility E-Motion & Chroma teams Inria Research Center Grenoble Rhône-Alpes Christian LAUGIER First Class Research Director at Inria San Francisco, 05/11/2015