Multimodal Interaction for Users with Autism in a 3D Educational Environment Ing. Alessandro Trivilini Prof. Licia Sbattella Ing. Roberto Tedesco 1
Screenshots 2
Screenshots 3
Introduction Existing Projects WEBminore project 3D educational environment University of Applied Sciences and Arts of Southern Switzerland ICF* project Extended version of the WHO ICF model MultiChancePoliTeam, Politecnico di Milano www.webminore.supsi.ch www.polimi.it Multimodal Interaction for Users with Autism in a 3d Educational Environment 4
Goals of the Project Define a new Multimodal Interaction Approach for Users with Autism in a 3D Educational Environment through easy iconic symbols and metaphors designed to guide them into the environment with a combination of visual, voice and textual modalities 5
Pillars of the Structure User Model Focused on attributes related with new technologies (mouse clicking, pointing, typing, etc) and on different profiles of autism. The project leverages ICF*, which is an extended version of the International Classification of Functioning, Disability and Health identifying right user attributes User Profiles Have been considered three specific profiles of autism. 1) a profile with high thinking and recognizing problems; 2) a profile with voice production and verbal communication problems; 3) a profile with social impairments and restricted patterns of behavior thinking, verbal communication and social impairment disorders Personalization The project takes into account the personalization of contents (exercised and educational activities), the input/ output interaction, the navigation in the 3D environment and the multimodality approach strictly related with the user profiles (text, voice and video) right content for right user Experimentation/Validation The prototype has been tested with six different users with six different levels if autistic disorder belonging to the three defined profiles, in collaboration of Esagramma, an Italian centre specialized in mental disorders and rehabilitation (www.esagramma.net) 6
Autism Main characteristics considered for the project Social impairments, communication difficulties Isolation from the surrounding environment (3D environment) Difficulty interpreting other thinking/feeling (collaborative environment) Difficulty to transform a communication s intent into a real message (embodiment) Cognitive impairment: when new visual information is coming from external world the brain has to elaborate it (interaction with the system)... and for these reasons, it was a good strategy to use the repetition approach of the same object (iconic fruit path) to use metaphors to aid memory, language and speech deficiencies 7
User Centred Approach Embody emotional experience in interaction Do not force users to get in touch with their emotions just simplifying their message or their graphical expressivity (through a GUI) Put attention on previous user experience and take care about his feelings with new technologies (how he looks at it, holds it, open or close it) All users should be able to understand immediately and intuitively what they can and cannot do within the system (avoiding invisible objects or automatic controls) 8
System Structure Wig-Wag Signs Fruith Path Mulimodal Interface Personalized Contents ICF* -> Static User Profile 3D Educational Environment 9
System Architecture USER PROFILE package GUI package Content package 10
ICF Attribute Classification Understanding the system interface Understanding interfaces, number of necessary steps to reach information, understanding metaphors (icons, buttons) Acting on the system interface Typing, using virtual keyboard, moving a pointer, mouse clicking ICF Standard Classification ICF* extension -> ITF Skills 11
User Profile Representation ICF s Body Function (26) Mental, voice, speech, seeing and hearing functions ICF s Activity Participation (34) Learning, thinking, reading, writing, tasks, communication functions Acting on the System Interface Attributes ICF s Rational Capabilities (12) Relationships, interpersonal interactions, community functions Understanding the system interface (4) Understanding interfaces, number of necessary steps to reach information, understanding metaphors (icons, buttons) Acting on the system interface (10) Typing, using virtual keyboard, moving a pointer, mouse clicking Understanding the System Interface Attributes Example of a User s Profile Disorder Representation Qualifiers range from [0-9] is used following the ICF guidelines, in order to shape the user s profiles [0] it will indicate that the problem impact is very low [4] it will indicate that the problem impact is very high 12
The User Profiles Acting on the System Interface Attributes Acting on the System Interface Attributes Acting on the System Interface Attributes Understanding the System Interface Attributes Understanding the System Interface Attributes Understanding the System Interface Attributes User profile 1 Access the 3D environment and simply navigate in it User profile 2 Access the 3D environment, find and read a newspaper User profile 3 Access the 3D environment and watch videos 13
Multimodal Interaction TV CONTENT NEWSPAPER CONTENT Multimodal Interaction Interfaces 14
Navigation Techniques Go to the left Go ahead Go to the right Guided Iconic (metaphor) Fruit Path Wig-Wag (metaphor) Sings Path 15
Importance of Perception Good Bad 16
Limits and Future Approach Limits Cognitive capabilities to use correctly mouse & keyboard Specific hardware interface and expensive software Future Approach Natural User Interface (NUI) Gestures Voice with motion sensor input device 17
Results & Constraints Results Fruit path and metaphors appreciated by most of the users 3D environment attractive for the users Multimodal interface well perceived by most of the users Wig-wag signs not completely followed by all users Constraints More personalized contents for different user profiles Keyboard and mouse as a technical limit 18
Conclusion Multidisciplinary project -> multidisciplinary experience Good approaches and techniques (metaphors) (Wig-Wag; Guided Fruit Path) Easy and understandable multimodal interface 3D world as a rich content environment for user with autism 19