VIDEO SURVEILLANCE AND BIOMEDICAL IMAGING Research Activities and Technology Transfer at PAVIS Samuele Martelli, Alessio Del Bue, Diego Sona, Vittorio Murino Istituto Italiano di Tecnologia (IIT), Genova name.lastname@iit.it 1. Introduction The Pattern Analysis and Computer Vision (PAVIS) Department focuses on activities related to the analysis and understanding of images, videos and patterns in general, performing cutting edge research in computer vision, pattern recognition and machine learning, as well as in multimodal data and sensor fusion, biometrics, multiview geometry and embedded computer vision. The research in PAVIS follows two main streams, namely video surveillance and biomedical imaging. We are particularly interested in exploring how social signals emerge in various conditions and how they can be used to design effective surveillance systems or to study behavioral pathologies. Indeed, we aim at discovering how findings from the human sciences (psychology, neuroscience, etc.) can be exploited to analyze situations occurring in a scene, in order to detect, recognize, understand and predict abnormal behaviors or conditions. To this end, we tackle typical issues such as detection, tracking, recognition, in other words, video analytics. We develop computational tools to obtain accurate description of the dynamics in videos from a camera or from a network of cameras, aiming at a resilience to light variations and occlusions, which are common aspects in video surveillance scenarios. Beyond standard vision sensors, we also deal with multiple and multimodal sensors, e.g., 3D sensors, microphones, RFID, infrared and thermal cameras, etc. This high load of information, however, leads to the necessity of processing most of the acquired data onboard, limiting the communication bandwidth. Hence, we are also addressing issues related to sensor networks, sensor fusion, and the design of embedded devices. Reliable surveillance systems must cope with biometrics issues as well, and in this context, we have research activities in non-cooperative face recognition at distance, gait recognition, and re-identification. The main objective of biomedical imaging stream is the development of methodologies allowing to investigate behavioral phenotyping at all levels,
2 ranging from the neuronal network connectivity to animals social behavior, through the analysis of brain connectivity, brain function and their behavioral correlate. To cope with these issues, we design models to integrate and investigate various sources of multimodal and multidimensional data acquired with various sensors, such as florescence microscopes, multi-electrodes arrays, biomedical instruments like magnetic resonance, etc. Most of the activities in this area aim at finding the morphological and physiological markers of neurological diseased (e.g., schizophrenia, Alzheimer, multiple sclerosis, autism, etc.) and how treatments affect the functions or the behavior. All PAVIS activities aim at designing intelligent systems, deserving special attention to applications to real problems, also involving other disciplines like neuroscience, psychology, biology, physics, and in cooperation with other research centers, universities, and both private and public partners. A strong mission of PAVIS is indeed to transfer the developed technology to the society, resulting in several industrial collaborations and projects. 2. Methodologies Background subtraction, object detection, object tracking, re-identification, and behavior analysis are the most important components of a video analytics system. PAVIS has some cutting edge technologies in this area, which exploit recent statistical and differential geometric theories and adapt them to challenging tasks that take place in real world scenarios. On the other side, research in Machine Learning at PAVIS focuses mainly on kernel methods. We have recently developed a general learning framework using vector-valued Reproducing Kernel Hilbert Spaces (RKHS) that allows us to simultaneously: (i) learn structured output; (ii) integrate input data from different modalities; and (iii) perform semi-supervised learning, using a small amount of labeled data. 3. Video Surveillance Social Signal Processing. In this activity we aim at modeling social signals, which encode the human attitude towards particular interactions and social environments. Facial expressions, body postures, vocal behaviors, how people wander in crowded spaces, are all examples of social signals. Beyond the identification of these signals, our main interest is to capture the hidden mechanisms that generate the signals, either generated autonomously or as a reaction to the behavior of another person. In social psychology, this is called Social Intelligence. To this aim we employ generative graphical models, inferring hidden causes underlying the visible observations. The current research focuses on the analysis of threatening behaviors for surveillance purposes, the study of the social dynamics that generate groups, and the evolution of dialogs.
Video Surveillance and Biomedical Imaging 3 Crowd Behavior Analysis. Research in video surveillance is shifting the attention from single person to groups monitoring and the analysis of their behavior. This level of abstraction provides event descriptions which are semantically more meaningful, highlighting barely visible relational connections among people. Automatic Crowd Analysis is a research area where we can investigate anomaly detection: panic scenarios, dangerous situations, illegal behaviors, etc. We are currently using a particle-based paradigm, in which a large set of virtual particles simulates the crowd behavior using visual cues such as the optical flow. Group Analysis. Analysis of small groups can be divided into group detection and group tracking. The objective of group detection is to find collections of people who share certain aspects and interact each other. To this end, we investigate on models embedding notions of social psychology, which offer novel research perspectives in video surveillance. On the other hand, group tracking consists in following tight formations of individuals while they are walking or interacting. One of the major difficulties lies in the high variability of groups seen as entities: splitting, merging, initialization and deletion are frequent events that characterize the life of a group, and that are usually modeled by heuristic rules, yielding limited generalization. Our idea is exploit the structure of data simultaneously tracking individuals and groups. Person Re-Identification. Person re-identification consists in recognizing individuals in diverse locations and times across different non-overlapping cameras. This task is fundamental for a set of surveillance applications, specially when dealing with large and structured environments such as museums, shopping malls, airports, convention centers, etc. As the typical video surveillance systems do not provide resolution high enough to work with facial or iris recognition, the classical solutions normally relies on appearance information, i.e., clothing and accessories. Nonetheless, the task is challenging because it must be robust to occlusions, changes in perspective, pose deformations, illumination variance etc. Hence, our activity is mainly focused on investigating new descriptors and methodologies to deal with these issues. Person Detection and Tracking. Detecting and tracking individuals in a given scene are very challenging tasks. We are interested in these issues for their implications in video surveillance and driver assistance systems. People detection is challenging because of the high variability of the body appearance. Detecting a person in crowded scenarios is even harder, due to the severe occlusions that may occur. Our research focuses on discriminatively learning a set of spatial relations between Poselet types. Tracking can be seen as an extension of people detection over time, generating the trajectories through the
4 video. Here we focus mainly on issues such as change of people appearance over time and occlusions. Visual Geometry Modeling. Modelling the visual appearance of natural and artificial objects is of paramount importance. Indeed, the automatic inference of the geometry of shapes solely from images might have a tremendous impact in many fields of science and engineering. In this regard, PAVIS is actively involved in 3D modeling at various scales, ranging from microscopy imaging where we push forward the boundaries of 3D super-resolution to a larger scale, where we are developing algorithms to geometrically self-localize sensors (microphones, cameras) freely deployed in an area and linked together by a network. We are also actively studying severely ill-posed problem such as the geometrical modelling of non-rigid shapes solely from 2D images. We have also developed efficient methods for accurately estimating the 3D shape of larger scale objects using Structure from Motion. 4. Biomedical Imaging Mice Behavior Analysis. PAVIS works on various aspects of social behavior analysis and understanding, and the application to small animals offers an interesting case-study. Indeed, mice behavior analysis plays a key role in neuroscience, allowing to study both the behavioral patterns of genetic models presenting human-like neurological diseases and the effect of pharmacotherapies on the behavioral impairments. We are therefore developing intelligent systems able to track multiple mice freely interacting in a cage and to automatically label their social behaviors based on video examples labeled by experts. In this framework, we are investigating and developing standard and deep models to infer behavioral patterns of groups of animals, which can characterize certain pathologies and the corresponding treatments. Structural and Functional Connectomics in Neuroimaging. Research in brain imaging is slowly shifting from the study of brain functions as located in specific areas to the investigation of the brain as a complex network of interacting functions. Thanks to the advent of noninvasive imaging tools like magnetic resonance it is nowadays possible to investigate the brain looking at the overall connectivity. In this imaging framework we are interested in understanding the structural and functional connectivity and their relationships, aiming at supporting neuroscientists and clinicians to characterize and/or recognize neurological and neurodegenerative pathologies (e.g., multiple sclerosis, schizophrenia, autism, etc.). Recently we focused our research on fiber bundles segmentation, discrimination of functional or structural connectivity between populations, characterization of brain modularity and joint analysis of structural connectivity and functions. Our long term goal is to investigate
Video Surveillance and Biomedical Imaging 5 how network changes are related to abnormal behavioral patterns objectively measured through behavior analysis systems. Multimodal Analysis of Neuronal Networks. Experiments with in-vitro neuronal networks allow to investigate fine network interactions and dynamics at the basis of the brain processing. This investigation however necessarily comes through the analysis and the understanding of the relationships between how connections between neurons are shaped and how the signal propagates in the networks. We aim at providing computational methods for a joint analysis of structural connectivity and function of neuronal network cultures, by integrating multimodal data consisting of images acquired with fluorescence microscopy and electrical activity recorded with high resolution Multi Electrodes Arrays. In particular, we are interested in image processing for cells segmentation and network reconstruction, identification of relationships between functional interactions and structural network, and interpretation of retina electrical activity under visual stimuli. 5. Technology Transfer During these years PAVIS gained a considerable experience in Machine Vision thanks to the numerous industrial collaborations (e.g., Omron, AVIO Aereo, Leonardo-Finmeccanica), ranging from visual inspection of complex 3D structures (e.g., aircraft engine components, car components, etc.) for detection of missing components, factory automation, robotized quality control through multispectral sensors, etc. All designed solutions are developed to have a technological impact and in synergy with supporting facilities in IIT they can be easily transferred to the productive system. Similarly, all solutions designed in other research activities are always aimed at a technological exploitation, like for example the DualCam project in surveillance. DualCam is an innovative security network camera which provides a visual stream overlapped with an acoustic image describing the sound landscape of a scene. This allows to detect and interpret events even in bad conditions like occlusions and variability of light and weather conditions.