Cistani, M., Vinciaelli, A., Segalin, C., and Peina, A. (2013) Unveiling the multimedia unconscious: implicit cognitive pocesses and multimedia content analysis. In: 21st ACM intenational confeence on Multimedia MM '13, 21-25 Oct 2013, Bacelona, Spain. Copyight 2013 The Authos A copy can be downloaded fo pesonal non-commecial eseach o study, without pio pemission o chage Content must not be changed in any way o epoduced in any fomat o medium without the fomal pemission of the copyight holde(s) When efeing to this wok, full bibliogaphic details must be given http://epints.gla.ac.uk/100502 Deposited on: 07 Januay 2015 Enlighten Reseach publications by membes of the Univesity of Glasgow http://epints.gla.ac.uk
Unveiling the Multimedia Unconscious: Implicit Cognitive Pocesses and Multimedia Content Analysis ABSTRACT Maco Cistani 1 Alessando Vinciaelli 2,3 1 Univesity of Veona (Italy) 2 Univesity of Glasgow (UK) maco.cistani@univ.it vincia@dcs.gla.ac.uk One of the main findings of cognitive sciences is that automatic pocesses of which we ae unawae shape, to a significant extent, ou peception of the envionment. The phenomenon applies not only to the eal wold, but also to multimedia data we consume evey day. Wheneve we look at pictues, watch a video o listen to audio ecodings, ou conscious attention effots focus on the obsevable content, but ou cognition spontaneously peceives intentions, beliefs, values, attitudes and othe constucts that, while being outside of ou conscious awaeness, still shape ou eactions and behavio. So fa, multimedia technologies have neglected such a phenomenon to a lage extent. This pape agues that taking into account cognitive effects is possible and it can also impove multimedia appoaches. As a suppoting poof-of-concept, the pape shows not only that thee ae visual pattens coelated with the pesonality taits of 300 Flick uses to a statistically significant extent, but also that the pesonality taits (both self-assessed and attibuted by othes) of those uses can be infeed fom the images these latte post as favouite. 1. INTRODUCTION Until a few yeas ago, poduction and diffusion of multimedia data equied skills and infastuctue that wee the pivilege of a few individuals and oganizations (achives, digital libaies, online epositoies, etc.) [38]. Nowadays, technologies as ubiquitous and use-fiendly as smatphones and tablets allow one to easily ceate multimedia mateial (pictues, videos, soundbites, text and thei combinations) and shae it with othes - typically though social media o othe online technologies - by simply pushing a button. In such a technological landscape, multimedia data is not just a way to tansmit knowledge and infomation - as it used to be taditionally fo any type of data [5, 41] - but one of the channels though which we inteact with othes. The coe idea of this aticle is that, in such an unpece- Pemission to make digital o had copies of all o pat of this wok fo pesonal o classoom use is ganted without fee povided that copies ae not made o distibuted fo pofit o commecial advantage and that copies bea this notice and the full citation on the fist page. To copy othewise, to epublish, to post on seves o to edistibute to lists, equies pio specific pemission and/o a fee. Copyight 20XX ACM X-XXXXX-XX-X/XX/XX...$15.00. Cistina Segalin 1 Alessando Peina 4 3 Idiap Reseach Institute (Switzeland) 4 Micosoft Reseach (USA) cistina.segalin@univ.it alpeina@micosoft.com dented scenaio, the exchange of multimedia data has become a fom of human-human communication and, theefoe, it should involve the cognitive phenomena typically obseved in human-human inteactions. This applies in paticula to implicit cognitive pocesses that take place outside ou conscious awaeness, but still shape to a lage extent ou peception of the wold and ou behavio [23], namely the tendency to expess and attibute to othes goals, values, intentions, taits, beliefs and any othe type of socially elevant chaacteistics [49]. To have a measue of how much multimedia data have become a means of communication between people, it is sufficient to conside a few statistics available on Youtube at the moment this aticle is being witten 1 : while uploading evey day 12 yeas of video mateial, Youtube uses access the popula on-line platfom one tillion times pe yea, an aveage of 140 visits pe peson on Eath (the figue efes to 2011). In othe wods, thee seems to be no multimedia sample poduced by one peson that is not consumed by someone else. Unlike a mee few yeas ago, ceation, diffusion and shaing of multimedia data is no longe the exclusive peogative of skilled pofessionals, but the eveyday pactice of the lay peson. Multimedia data ae no longe, o no longe exclusively, the caefully cafted poduct of ceativity and communication skills, but the spontaneous expession of common individuals involved in eveyday social inteactions. The poblem is that ou cognitive pocesses ae the esult of a long evolutionay histoy and cannot change at the pace of technology. Theefoe, ou cognition keeps following pattens that wee shaped duing times when technology was fa fom existing [32]. In paticula, a lage body of evidence shows that ou cognition constantly woks to make sense of the wold aound us and that this happens, to a lage extent, effotlessly, and even unintentionally [48]. This means that the infomation we gathe and pocess though ou conscious attention - the typical ealm of cuent multimedia technologies - is only one of the factos that dive ou eactions towads the envionment, the othes being implicit, even automatic pocesses: implicit attitudes, infeences, goals and theoies, and the affect and behavios they poduce [49], whee the wod implicit means outside ou conscious awaeness. To the best of ou knowledge, multimedia appoaches neglected so fa to a lage, if not full, extent the phenomena above. Most of the cuent technologies take into account 1 http://www.youtube.com/yt/pess/statistics.html
only infomation that can be automatically detected in the data (e.g., objects in pictues) o infeed fom it (e.g., gene fom music). The few attempts to take into account implicit cognitive pocesses focused on obsevable effects, including emotional, behavioal and physiological eactions, used, e.g., fo etieval [1] and tagging puposes [36]. Howeve, such eactions might be difficult to detect, especially in settings whee social noms impose behavioal limitations (e.g., public spaces). Futhemoe, obsevable eactions ae nothing else than effects that follow the actual changes in the use, namely those that concen implicit attitudes, infeences, goals and theoies (see above). The possible solution to such a state of affais is that cognitive changes behind obsevable eactions ae not andom, but tend to follow, accoding to the Bunswik Lens [4], stable and pedictable pattens. The Bunswik Lens, one of the most effective models developed in cognitive psychology, povides a famewok suitable fo investigating how multimedia data can be adopted as an obsevable evidence of attitudes, infeences, goals and theoies (see above) of data poduces. Symmetically, the model helps to explain how data consumes attibute attitudes, infeences, goals and theoies to data poduces. This pape shows that cognitive effects ae detectable at least in the case of the inteplay between Flick pictues and pesonality taits of Flick uses. The esults, obtained ove PsychoFlick (a novel image dataset of 60,000 images posted by 300 individuals), show not only that thee is a statistically significant coelation between pesonality taits of the uses and featues extacted fom the images they post, but also that the same featues ae coelated with the taits that pictue obseves assign to the uses, even if obseves and uses have neve been in contact. Futhemoe, both associations ae sufficiently stable to be leaned by supevised statistical classifies. This opens up to a set of applications like, e.g., automatic attibute pediction: given a pool of images, the goal is to infe pesonal chaacteistics of its owne. This goal is pefomed hee by pojecting images on low-dimensional manifolds and exploiting spase egession. The est of this pape is oganized as follows: Section2 suveys eseach tends elevant to this wok, Section 3 descibes the Bunswik Lens model, Section 4 pesents the dataset used fo the expeiments, Section 5 epots on expeimental evidence suppoting the coe-idea of this pape, Section 6 pesents application domains that can benefit fom this wok and the final Section 7 daws some conclusions. 2. NEIGHBORING AREAS The key-idea of this aticle is that the exchange of multimedia data has become a fom of human-human communication and, theefoe, it should give ise to the same cognitive phenomena (e.g., see [48, 49]) typically obseved in any human-human inteaction. To the best of ou knowledge, this aticle is the fist attempt to adopt such a pespective in multimedia technologies. Howeve, seveal domains conside neighboing issues that, while being diffeent fom the ones poposed in this aticle, still include aspects elevant to this wok. The application of the sociotechnical pespective in studying the use of digital libaies - until a few yeas ago the most common infastuctue fo the exchange of multimedia data - is one of the ealiest attempts to take into account social issues in technological applications: To undestand, use, plan fo and evaluate digital libaies, we need to attend to social pactice, which we define as people s outine activities that ae leaned, shaped, and pefomed individually and togethe [38]. The main diffeence between the sociotechnical pespective and the eseach diection poposed in this wok is that the fome focuses on use and usability issues (especially in pofessional and institutional settings) while the latte tagets the communication between individuals, a step made possible only by ecent technologies (social media, mobile devices, etc.). In paallel, seveal effots wee done to impove multimedia technologies by automatically detecting and undestanding emotional, behavioal and physiological eactions of data consumes (e.g., if a peson watching a video laughs, then the video can be tagged as funny ) [1, 25, 36]. The coe-idea of these tends is that the content of the data poduces obsevable changes in data consumes, then the obsevation of these latte povides infomation about the data. The main diffeence with espect to this aticle is that the accent is on the data content, like in most of the multimedia technologies, and not on the communication pocess undelying the data exchange between individuals. Moe ecently, some woks investigated the inteplay between obsevable chaacteistics of multimedia data and cognition [26, 54]. The fist wok [26] consides images tagged as favouite by a cetain peson as an expession of he aesthetic pefeences and shows that, given a cetain amount of pictues tagged as favouite by a cetain individual, it is possible to pedict whethe the same will happen fo anothe pictue o set of pictues. The second wok [54] investigates the chaacteistics of abstact paintings that stimulate cetain emotional eactions athe than othes. Both woks shift the attention fom the bae content of images to thei potential ole in a communication pocess, namely a peson expessing aesthetic pefeences in [26] and a painte eliciting emotions in [54]. Howeve, unlike the pespective advocated in this aticle, both woks take into account only one of the paties involved in the communication pocess. To the best of ou knowledge, the only two woks that seem to conside multimedia data as a fom of communication ae in [11, 14]. The wok in [14] studies the peception of pofile pictues on social media and, in paticula, the ageement between the actual pesonality taits of pofile holdes and taits attibuted by othes based on the pofile pictue. The wok in [11], does a simila analysis, but it consides all elements that can appea in a pofile. Not supisingly, these woks focus on social media, an inteaction-oiented technology that allow uses to use multimedia mateial to communicate with othes. Howeve ealy, the appoaches in [11, 14] seem to confim the action of implicit cognitive pocesses when using multimedia data in a communication scenaio, the key-idea advocated in this aticle. Still, both woks focus on a specific case and do not ty to identify the undelying pespective that can be applied to many diffeent cases. 3. THE MULTIMEDIA LENS MODEL This section povides a conceptual famewok that illustates the pespective poposed in this aticle, namely a simplified vesion of the Bunswik s Lens (see Figue 1), the model oiginally poposed in [4] and successively modified to investigate, among othe inteaction phenomena, the in-
Data Poduce Extenalization State μ S measue ρ EV Ecological Validity featue 1 featue 2 featue 3 ρ Functional FV Validity featue N-2 featue N-1 featue N Attibution μ P Peceptual Judgment measue ρ RV Repesentation Validity Data Consume Figue 1: The pictue shows a simplified vesion of the Bunswik Lens Model adapted to the exchange of multimedia data between a Data Poduce and a Data Consume. fluence of nonvebal behavio in face-to-face inteactions [43] o the judgment of appot [2]. In the model of Figue 1, the multimedia data is consideed a fom of communication between Data Poduces (DP) and Data Consumes (DC). The key-idea of this aticle is that the pocess includes not only the exchange of content, a poblem that the multimedia indexing and etieval community has extensively investigated fo at least two decades, but also implicit cognitive pocesses typical of any human-human inteaction like, e.g., the spontaneous attibution of socially elevant chaacteistics (attactiveness, tustwothiness, etc.) o the development of impessions. The DP is always assumed to be in a cetain state that can be eithe tansient (e.g., emotions, attitudes, goals, physiological conditions etc.) o stable (e.g., pesonality taits, values, social status, etc.). In opeational tems, the states ae defined as quantitative measues (identified as µ S in Figue 1) to be obtained via objective pocesses depending on the paticula case unde obsevation. Fo example, in the case of the social status, the measue can be the yealy income of the DP, while fo the physiological condition it can be the heat ate o the galvanic skin conductance. In many cases, the states coespond to psychological constucts (e.g., pesonality taits o intepesonal attactiveness) and the measues ae the outcome of psychometic questionnaies. These latte ae typically administeed to the DPs and include questions associated to Liket scales (see Section 5 fo an example). Accoding to the model, the multimedia data ae an extenalization of the DP state, i.e. an obsevable effect of it. Futhemoe, the data is all the DCs know about a DP. Fom an opeational point of view, the data coespond not only to the actual multimedia mateial (e.g., pictues, video, soundbites, etc.), but also to any featue that can be extacted, manually o automatically, fom the mateial itself. The empiical covaiation of state measues and featues quantifies the ecological validity of these latte, i.e. thei effectiveness in accounting fo the DP state. In Figue 1, the ecological validity is indicated with ρ EV and, typically, it coesponds to the coelation o the Speaman coefficient between featues and µ S. When the DCs consume the data, they attibute the DP a state of measue µ P. The pocess is called attibution and µ P is efeed to as peceptual judgment. Fo example, the DCs can attibute a cetain yealy income to the DP based on the pictues and videos this latte shows. In the case of the psychological constucts (see Section 5), the attibution pocess is typically unconscious and it takes place spontaneously, whethe the DC needs it (wants it) o not [48, 49]. In pinciple, µ S and µ P should have the same value (o at least simila values), but communication pocesses ae always noisy, especially when the communication takes place though ambiguous channels like multimedia data ae. The empiical covaiation between featues and peceptual judgments accounts fo the epesentation validity of the featues (identified as ρ RV ), i.e. fo the influence these latte have on the attibution pocess. Like in the case of ρ EV, the most common measuements of ρ RV ae coelation and Speaman coefficient. The cognitive pocesses this pape focuses on ae active in paticula at the peceptual judgment stage, when DCs unconsciously develop an impession about the DP even if all they know about this latte is the multimedia data they ae consuming. Howeve, the pocesses ae impotant fo the DP as well because, in a communication scenaio, thee is no data poduction without an attempt to convey an impession, i.e. to ensue that µ S and µ P ae close to each othe. The empiical covaiation of µ S and µ P (identified as ρ F V in Figue 1) accounts fo the latte aspect and it is called functional validity. 4. THE PSYCHOFLICKR DATASET We expeimented the coe-idea of this pape on Flick 2, one of the most popula online photo-shaing platfoms. To this pupose we collected a copus, dubbed PsychoFlick, that eflects the Lens Model and includes both pictues and pesonality assessments. The copus is publicly available at [the dataset will be made available in case of acceptance] and was collected as follows: we contacted 300 andom po uses, i.e. individuals that pay a yealy fee in ode to access pivileged Flick functionalities. These uses ae expected to be, on aveage, moe adept than othes to photogaphy language and techniques. Fo each of these 300 uses, we collected the 200 latest pictues made by othes they labeled as favouite, fo a total of 60,000 images. Futhemoe, each use filled the BFI-10 (Big Five Inventoy 10) [39], a 2 www.flick.com.
Extenalization Data Poduce Openness Conscientiousness 0.14-0.15-0.14-0.13-0.13 Size Regions Hue Ang. Disp. Textue L1 ch.h Textue L3 ch.v -0.13 # Faces -0.27 0.12 Dominance 0.31 State μ S -0.13 Colofulness -0.24 μ P Extavesion 0.12 0.12 0.16-0.13 # People Size People # Faces # Edges 0.27-0.28-0.15-0.19-0.13 0.52 0.40 0.46-0.12 Openness Conscientiousness Peceptual Judgment Extavesion Attibution Data Consume Ageeableness 0.15 # Cas 0.17 Ageeableness Neuoticism -0.13 Puple -0.25 Neuoticism Figue 2: The pictue shows the Bunswik Lens model fo the PsychoFlick dataset, whee the state coesponds to the Big Five taits (as pe assessed with the BFI-10). Ecological and Repesentation validities ae measued with the Speaman Coefficient and the pictue shows (fo each tait) featues fo which both values ae statistically significant (p < 5%). pesonality questionnaie aimed at measuing the pesonality of an individual in tems of the Big Five, five boad dimensions shown to captue most of the individual diffeences [42]. The outcome of the BFI-10 is a five dimensional vecto whee each component measues how high an individual is with espect to each of the Big Five taits, namely Openness (tendency to be intellectually open, cuious and have wide inteests), Conscientiousness (tendency to be esponsible, eliable and tustwothy), Extavesion (tendency to inteact and spend time with othes), Ageeableness (tendency to be kind, geneous, etc.) and Neuoticism (tendency to expeience the negative aspects of life, to be anxious, sensitive, etc.). In paticula, fo each tait, we have an intege which goes fom -4 (low tendency) to 4 (high tendency). Finally, we hied eight assessos that looked at the set of 200 favoite images povided by each of the uses and, fo each of them, filled the BFI-10 questionnaie. Howeve, while the Flick uses adopted the self-assessment vesion of the BFI-10, the assessos used the othe-assessment vesion. In othe wods, the uses ated statements like I am a eseved peson, while the assessos ated statements like This peson is eseved, whee by peson is meant the use that has labeled the images unde exam as favouite. Each of the 8 assessos filled the pesonality questionnaies fo each of the 300 uses. The uses and the assessos wee neve in contact and, futhemoe, the images wee the only infomation the assessos had at disposition about the uses unde exam. The 8 pesonality atings poduced by the diffeent assessos about the same use wee aveaged to obtain the peceptual judgment (accoding to expeimental psychology pactices [39]). The ageement among the assessos was measued with the Kippendoff s α [22], a eliability coefficient suitable O C E A N α 0.07 0.16 0.23 0.19 0.20 Table 1: Kippendoff s α fo the Big Five taits. fo a wide vaiety of assessments (binay, nominal, odinal, inteval etc.), and obust to small sample sizes. Table 1 epots the α values fo the diffeent taits. The values ae statistically significant and compaable to those obseved in the liteatue fo zeo acquaintance scenaios [3], i.e. situations whee assessos and subjects being ated do not have any pesonal contact (like it in the PsychoFlick copus). In tems of the Lens Model, the po uses ae the Data Poduces, the assessos ae the Data Consumes, the pesonality is the state and the outcome of the BFI questionnaie is the state measue (see Section 3 fo moe details). 5. MULTIMEDIA LENS AND FLICKR This section measues, in quantitative tems, whethe implicit cognitive pocesses ae actually at wok in the scenaio undelying the PsychoFlick copus o not. In paticula, the section addesses the following questions: Is thee a consistent elation between featues extacted fom sets of favouite images and pesonality taits of Flick uses (both self-assessed and attibuted)? If yes, is the elation sufficiently stable to automatically pedict the pesonality taits of Flick uses (both self-assessed and attibuted) based on sets of favouite images? If the key-idea of this wok holds, and implicit cognitive pocesses influence the exchange of multimedia data (accoding
Categoy Name L Shot Desciption Use of light 1 Aveage pixel intensity of V channel [9] HSV statistics 4 Mean of S channel and standad deviation of S, V channels [27]; Hue angula dispesion in IHLS colo space [30] Emotion-based 3 Amount of Pleasue, Aousal, Dominance [27, 50] Colofulness 1 Colofulness measue based on Eath Move s Distance (EMD) [9, 27] Colo Name 11 Amount of Black,, Bown, Geen, Gay, Oange, Pink, Puple,, White, Yellow [27] Entopy 1 Image entopy [26] Aesthetics Wavelet textues 12 Level of spatial gaininess measued with a thee-level (L1,L2,L3) Daubechies wavelet tansfom on the HSV channels [9] Tamua 3 Amount of Coaseness, Contast, Diectionality [46] GLCM-featues 12 Amount of Contast, Coelation, Enegy, Homogeneity fo each HSV channel [27] Edges 1 Total numbe (#) of edge points, extacted with Canny [26] Level of detail 1 Numbe of egions (afte mean shift segmentation) [6, 16] Regions 1 Aveage size of the egions (afte mean shift segmentation) [6, 16] Low depth of field (DOF) 3 Amount of focus shapness in the inne pat of the image w..t. the oveall focus [9, 27] Rule of thids 2 Mean of S,V channels in the inne ectangle of the image [9, 27] Image paametes 2 Size of the image [9, 26] Objects 28 Objects detectos [12]: we kept the numbe of instances (#) and thei aveage bounding box size Content Faces 2 Numbe (#) and size of faces afte Viola-Jones face detection algoithm [51] GIST desciptos 24 Level of openness, uggedness, oughness and expansion fo scene ecognition [35]. Table 2: Summay of all featues. The column L indicates the featue vecto length fo each type of featue. to the Lens Model), then the answe should be positive in both cases. 5.1 Featues and Pesonality We adopted a wide, though not exhaustive, spectum of featues, hee gouped into two families (see Table 2). On one side, we have the cues that focus on aesthetic aspects [9, 27]: the eason is that the PsychoFlick copus includes pictues posted as favouite, i.e. likely to epesent the aesthetic pefeences of the uses unde exam. On the othe side, we focused on the content of the images; to this end, we employed obust pobabilistic object detectos [12] (fo a complete list of all detectable objects see [12]); we also etained the aveage aea (the algoithm gives also the bounding box of the detected objects). In addition, we focused on the faces, adopting the standad Viola-Jones face detection algoithm [51] implemented in the OpenCV libaies. Finally, we adopted the GIST scene desciptos, which amounts to apply a set of oiented band-pass filtes. Figue 2 shows the featues with highe ecological (covaiation of self-assessment and featues) and epesentation (covaiation of featues and peceptual judgment) validity with espect to the Big Five taits. The covaiations, measued with the Speaman Coefficient, ae statistically significant (p < 5%). Theefoe, implicit cognitive pocesses seem to be actually at wok when Flick uses shae thei set of favouite images. The answe to the fist question at the beginning of this section is positive. Futhe confimation comes fom Figue 3, showing a andom selection of images labeled as favouite by extavet (collage a ) and intovet (collage b ) subjects. The fome appea to pictue people way moe fequently than intovet ones (80% and 17% of the images in the collage, espectively). 5.2 Pesonality Pediction Ecological and epesentation validity values of Figue 2 seem to suggest that pedicting pesonality taits (both selfassessed and attibuted) using favouite images is possible. The poblem was cast as a egession instance on the taits of the uses, consideing uses as the sets of thei pefeed images (see appendix A fo a desciption of the egession appoach). The pefomance was measued with the Speaman coelation coefficient between actual and pedicted pesonality taits, the highe the coefficient, the close the pediction to the tue value. The esults ae epoted in Table 3 whee the fist column shows the tait, the second explains whethe the pedicted tait is self-assessed o attibuted by the assessos, Max ρ is the maximum coelation found acoss the tests (i.e. the vaious configuations of the egession appoach), Mean (Std) ae mean value and standad deviation computed on coelations with p-values < 5%, and % s.s is the pecentage of diffeent configuations of the egession appoach that esulted in a statistically significant esult. In line with the liteatue on pesonality computing [28], the pefomances achieved ove self-assessed taits ae lowe than those obtained ove attibuted ones. The eason is that the fome depend on an individual assessment and tend to be moe noisy while the latte, esulting fom the consensus among diffeent assessos, tend to coelate bette with measuable chaacteistics of the data. In paticula, fo the attibuted taits, all configuations of the egession appoach tested in the expeiments led to statistically significant esults, while fo the self-assessed taits this happens only fo Openness. The best pefomance is achieved, fo the attibuted assessments, fo Extavesion and Conscientiousness. The same applies to most of the woks on pesonality computing pesented in the liteatue and the eason is that Extavesion and Conscientiousness ae the two taits people peceive moe quickly and effectively [19]. In the case of this wok, the pefomance is satisfactoy fo Neuoticism and Ageeableness as well. The tait fo which the pefomance is lowe is Openness. The pobable explanation is that the distibution of the uses along such tait is peaked aound high values (Openness is the tait of ceativity and po Flick uses ae, not supisingly, highe than the aveage along the tait). The esults pesented in Table 3 ae statistically significant (p < 5%) and, theefoe, the answe to the second question of the beginning of this section is positive. In othe wods, implicit cognitive pocesses seem to influence the attibution of pesonality taits in Data Consumes watching favouite pictues on Flick.
(a) Figue 3: Collage (a) and (b) ae a andom selection of favouite pictues of subjects high and low in Extavesion (as pe attibuted by the assessos), espectively. The most impotant diffeence is that extovet individuals show a pefeence fo pictues potaying people (80% of the samples in the collage) while intovet show the opposite pefeence (17% of the pictues in the collage). (b) Tait Label Max ρ Mean (Std) ρ % s.s. O Self 0.25 0.17 (0.04) 100% Attibuted 0.35 0.32 (0.04) 100% C Self 0.24 0.22 (0.03) 44% Attibuted 0.57 0.49 (0.05) 100% E Self 0.28 0.19 (0.05) 88% Attibuted 0.62 0.55 (0.03) 100% A Self 0.20 0.17 (0.03) 55% Attibuted 0.52 0.45 (0.05) 100% N Self 0.14 0.12 (0.07) 7% Attibuted 0.60 0.54 (0.04) 100% Table 3: Pediction Results. ρ is the Speaman Coelation Coefficient 6. POTENTIAL APPLICATIONS Section 2 suveys aeas that shae some aspects with the pespective poposed in this aticle while still showing substantial diffeences. This section focuses on application domains that involve the exchange of multimedia data and, theefoe, might benefit fom taking into account the cognitive pocesses that, accoding to the esults pesented above, seem to influence such type of pocess. Undestanding and modeling of cognitive pocesses involved in multimedia data consumption ae likely to be beneficial fo Human Infomation Inteaction (HII), the domain studying the elationship between people and infomation [13]. HII eseaches ae paticulaly inteested in modelling people poflections, i.e. individual s conscious and unconscious pojections on infomation objects (e.g., pictues) and the eflections that othe people and machines ceate to those pojections (e.g., links and annotations) [29]. This applies in paticula to multimedia etieval technologies that might be enhanced by taking into account not only the data content (like most of cuent technologies do [25]), but also the inteplay between content and peceptual judgments (pesonality, values, goals, intentions, etc.). The ole of cognitive biases can be of inteest fo Digital Humanities as well, especially fo what concens the effot towads new modes of knowledge fomation enabled by netwoked, digital envionments and the focus on distinctive modes of poducing knowledge and distinctive models of knowledge itself [5]. In paticula, Digital Humanities investigate the impact of media authoing technologies on the tansmission of knowledge and infomation, a phenomenon likely to involve implicit cognitive pocesses like those descibed in this wok. In a simila vein, the pespective adopted in this aticle can be useful in Big Data Analytics - the domain aimed at making sense of lage amounts of unstuctued data [31] - one of the most impotant challenges technology faces today. In paticula, thee is consensus among Big Data expets that no useful infomation can be extacted fom lage databases without associating automatic mining appoaches and human intepetation [34]. This latte is likely to be influenced by cognitive pocesses simila to those illustated in the expeiments of this wok. Vial maketing, the diffusion of infomation about the poduct and its adoption ove the netwok [24], is an advetisement technique aimed at speading infomation as widely as possible though (mostly online) wod of mouth mechanisms. The pevious pat of this pape has shown that the exchange of multimedia data, being a fom of humanhuman communication, can be thought of as a fom of wod of mouth. Theefoe, implicit cognitive pocesses might contibute to explain and enhance viality. In the same vein, communication stategies based on social media can benefit fom the pediction of peceptual judgments likely to be attibuted to a given multimedia message diffused though online social platfoms [20]. Accoding to the Euopean Consume Commissionee, Pesonal data is the new oil of the intenet and the new cuency of the digital wold [15]. The states of data poduces (see Section 3) often coespond to pesonal chaacteistics of potential inteest fo diffeent bodies (e.g., companies tying to model thei customes o govenments inteested in gatheing infomation about the population). Appoaches like those pesented in this wok can help to obtain such infomation by analyzing publicly available data that people usually post on pesonal home pages, Youtube, Facebook, etc. [21]. In paallel, the development of technologies ca-
pable of going beyond the mee content and infe pesonal chaacteistics of data poduces equie a edefinition of the concept of pivacy and a caeful analysis of ethical issues [8]. A peculia fom of communication though multimedia mateial is the paticipation in online games whee seveal paticipants inteact via avatas o animated chaactes. The choice of a paticula chaacte o paticula gaming stategies and options is likely to convey infomation about the playe states (see, e.g., the appoaches in [17, 53, 55] fo the case of pesonality). In a simila way, compute mediated communication can be influenced by implicit cognitive pocesses via inteface chaacteistics like the pofile pictue of Skype uses. Ceating and viewing photogaphs as a pocess of selfinsight and pesonal change is the main pinciple of phototheapy and theapeutic photogaphy [52, 45], two ecent psychology pespectives; fo the theapists, Images povide an undecuent of emotion and ideas that enich intepesonal dynamics, often on a level that is not fully conscious o capable of being vebalized. Of paticula inteest fo these fields is how the language of composition and visual design intesects with the language of unconscious pimay cognitive pocesses, including emotional/ideational association. Ou study suggests that answes to these questions may be found with the help of computes. Last, but not least, it is appaent the cucial ole that image pocessing and machine leaning would have; at the same time, ou study delineates new challenges fo these aeas; fo example, discoveing visual pattens that coelate with pesonal taits in a stonge way than odinay featues could be a eseach mission fo the field of deep leaning and featue leaning [44]. Geneative modeling can also be involved, looking fo new models that mimic the way divese visual featues should be combined togethe to communicate a cetain pesonal tait. An immediate example applies to the Counting Gid used in Section A.1: in this pape, we employed CG as a mee dimensionality eduction stategy, without accounting the taits label. Including this infomation may lead to a low-dimensional embedding whee neaby images exhibit simila featues and pesonal taits. The list pesented in this section is fa fom being exhaustive, but it is epesentative of the scenaios whee the investigations poposed in this aticle can be elevant, namely those whee individuals poduce, exchange and consume (possibily multimedia) data. 7. CONCLUSIONS This aticle advocates the idea that the exchange of multimedia data has become a human-human communication scenaio and, theefoe, it involves the same cognitive phenomena of any othe fom of inteaction between people, especially when it comes to expession and mutual attibution of socially elevant chaacteistics (attactiveness, social status, pesonality, goals, values, intentions, etc.). As a suppoting evidence, the pape poposes expeiments on the inteplay between pesonality taits and Flick pictues. The esults show that the pesonality of an individual can be pedicted, to a statistically significant extent, though the pictues she labels as favouite. Futhemoe, the expeiments show that the images can be used to pedict the taits that othes attibute to such an individual. Theefoe, at least fo what concens pesonality, the exchange of images via Flick seems to wok accoding to the Bunswik Lens (see Section 3), the cognitive model undelying social inteactions. In othe wods, the key-idea poposed in this wok appeas to hold. To the best of ou knowledge, such a pespective has neve been adopted in a multimedia technology context befoe. The pobable eason is that multimedia data became an inteaction channel only ecently, when the diffusion of appopiate technologies fo data poduction (cameas, smatphones, tablets, etc.) and consumption (social media, digital libaies, etc.) made it possible to exchange multimedia data as easily as we peviously exchanged witten mateial (lettes, messages, etc.) [5]. This new scenaio opens seveal eseach questions (the list is not exhaustive): Is it possible to impove multimedia technologies by taking into account implicit cognitive pocesses? Do implicit cognitive pocesses influence ou behavio as multimedia technology uses? Does multimedia technology need to change to accomodate implicit cognitive pocesses? If yes, how? What do we eveal about ouselves when we shae multimedia data? What is the effect of the multimedia data we shae on the impession othes develop about us? It can be expected that the peceptual judgments we make about those who poduce the data we consume end up influencing ou peception of the data. Fo example, we might tend to like moe o to find moe elevant data poduced by people we peceive as moe simila to us. If actually obseved, such an effect (known in psychology as similaityattaction [7]), might not only impove etieval technologies, but also contibute to explain ou behavio as uses and, in ultimate analysis, lead to highe technology usability and effectiveness. Simila consideations apply to any technology that involves the consumption of data. Symmetically, the inceasing amount of multimedia infomation we poduce and shae (Instagam pictues, Tweets including pointes to video and audio data, etc.) is pobably contibuting to a lage and lage extent to ou appeaance, one of the chaacteistics that influence most the impession othes develop about us (an effect known as the halo-effect psychology [33]). Howeve, while we know how to manage ou appeaance in face-to-face inteactions, in most cases we ae still not awae of the way othes see us though the lens of the multimedia data we poduce. The two examples above show how cognitive and technological issues ae tightly intetwined in eveyday scenaios involving poduction and consumption of multimedia data. The two cases focus on specific aspects, but the pespective poposed in this wok might show that the full ange of phenomena taking place in face-to-face inteactions (see [10] fo a monogaph) take place though multimedia data as well. If tue, the doo would be open towads new multimedia applications as well as novel findings in cognitive sciences. APPENDIX A. THE REGRESSION APPROACH To apply a standad egession appoach is poblematic because thee ae multiple images associated to the same
taget. Staightfowad algoithms like, e.g., summing all the image desciptos of each use, and then pefom egession, does not wok because such pocess adds noise to a weak signal. Multiple instance egession [40] is also unadvisable because of its high computational complexity, especially when the numbe of images fo each use is lage. Theefoe, we popose an altenative appoach composed by the following thee steps: e H.edge d H. e edge d H. edge H. edge e d e d H. e edge d H. edge e d H. ed edge 1. Featue Extaction and Nomalization. We fist extact fom all the images the set of featues listed in Section 5.1; since each z-th cue expesses the level of pesence of a given quantity, i.e. a count c z, we can think each image as an histogam of counts {c z}, o bag-of-featues (BoF). Afte that, we nomalize each c z to ensue that each featue takes values in the same ange. This avoids some featues (e.g., numbe of edges) to ovecome othes (e.g., GIST, amount of coaseness) 2. Clusteing. Afte dividing the uses in taining and testing uses, we conside all the images of the taining uses. By means of a clusteing algoithm, we lean a low-dimensional epesentation that maps each t-th image (i.e. its BoF) in a 2-dimensional location l t, lying on a smooth manifold. As clusteing method we employed the Counting Gid [37], a ecent geneative model which embeds BoF epesentations in N-dimensional manifolds 3. This way, each use u becomes a set of locations L u = {l t } on the manifold. 3. Regession and Tait Pediction. Consideing the taining uses, we tain a egesso to the pesonality taits. In specific, fo each use u we have a fivedimensional taget that chaacteizes the Big Five pesonality taits p {O, C, E, A, N}, whee each tait is descibed by a value y u p [ 4, 4]. As egession method, we used Lasso [47]. Tait pediction amounts to test the egesso on the test uses. In the following, we will detail the latte two steps of the pocess. A.1 Clusteing: the Counting Gid Model The counting gid (CG) is a geneative model ecently intoduced in [37] fo analyzing images collections. It assumes that images ae epesented as histogams {c z} o bags of featues, whee c z counts the occuences of featue z. Consideing its two-dimensional vesion, a CG is a 2D finite discete gid whee each location i = (x, y) contains a nomalized count of featues π i,z. Unde this model, an image (i.e. its BoF {c z}) could be thought as poduced by the following geneative pocess: a small window is located in the gid, aveaging the featue counts within it to obtain a local pobability mass function ove the featues, and then geneating fom it an appopiate numbe of featues in the bag (see Fig. 4). In othe wods, unlike a staightfowad embedding (e.g. PCA) that links an image with a point location, the counting gid foces the image to link with a small window of locations. Given that the size E 1 E 2 of 3 Hee we decide N = 2 fo the sake of claity; othe dimensions can be exploed. In addition, we tied diffeent dimensionality eduction appoaches (Mixtues of Diichlet distibutions), leading to infeio pefomances. H. edge e d H. edge e d H. edge Figue 4: Geneating an image fom a simple 3 3 counting gid: given a 2 2 window on the gid, we aveage the featue counts, obtaining a bag of featues which coesponds to the final image. V. edge and H. edge ae toy featues meaning vetical and hoizontal edges, espectively. a counting gid is usually small compaed to the numbe of images, this also foces windows linked to diffeent images to ovelap, and to co-exist by finding a shaed compomise in the featue counts located in thei intesection. The oveall effect of these constaints is to poduce locally smooth tansitions between stongly diffeent featue counts by gadually phasing featues in/out in the intemediate locations. In pactice, local neighbohoods in the gid epesent simila concepts and images mapped in close locations ae somehow simila. Fomally, the counting gid π i,z is a 2D finite discete gid, spatially indexed by i = (x, y) [1... E 1] [1... E 2], and containing nomalized counts of featues indexed by z. Thus, we have z π i,z = 1 eveywhee on the gid. A given BoF {c z} is geneated by selecting a cetain location k, calculating the distibution h k,z = 1 W n i W k π i,z by aveaging all the wods counts within the window W k (with aea W n) that stats at k, and then dawing featues counts fom this distibution. In othe wods, the position of the window k in the gid is a latent vaiable; given k, the likelihood of {c z} is p({c z} k) = z (h k,z ) cz = α z e d ( i W k π i,z ) cz, (1) whee α is a fixed nomalization facto. To lean a counting gid, we need to maximize the likelihood ove all taining images T, that can be witten as p({{c t z}, k t } T t=1) ( ) c t z π i,z, (2) t z i W k t which is intactable, much like in mixtues; theefoe, it is necessay to employ an iteative EM algoithm. Stating fom a andom initialization of the counting gid π, the E- step aligns all bags of featues to gid windows, to match the bags histogams, infeing whee each bag maps on the gid, i.e. q t (i) exp z c t z log h i,z (3) In the M-step the model paamete, i.e. the counting gid π, is e-estimated. Fo details on the leaning algoithm and on its efficiency, the eade can efe to the oiginal papes [37, 18]. Fo ou puposes, the most inteesting outputs ae the
posteio pobabilities q t s, the position in the gid of each image. Summing ove the entie gid the contibutes q t (i), which ae due to the images of a use, povides a signatue L u. Essentially, it is a 2D matix, of the same dimension of the gid, whee some locations {i} ae weighted by the q t s, indicating that in such locations thee ae some images of the use u. A.2 Regession and Tait Pediction To assess the validity of ou pediction method, we used the Leave-One-Use-Out paadigm. We consideed CGs of vaious complexities with size E = [20 20, 25 25,... 65 65] and window W = [5 5] and we leant a model with all the images belonging to the taining uses. Then, we computed L u fo each use, and used this epesentation to egess on the pofile yp u. We leaned the egession weight vecto w by minimizing the eo function E(w) = L i=1 ( y p w T L u (i)) 2 (4) whee L indicates the numbe positions in the gid. We solved the poblem using Lasso [47], a shinkage and selection method fo linea egession which enfoces the spasity on coefficients w by bounding the sum of the absolute values of the coefficients. The bound is a constaint that has to be taken into account when minimizing the eo function. At this point the taining phase is completed and to pedict the pesonality tain of the held out (test) use, we 1) infe the mappings of its images on the counting gid (Eq.3), 2) compute is latent desciption L test (i) and 3) by calculating we pedict the tait. B. REFERENCES ŷ test p = w T L test, (5) [1] I. Aapakis, J.M. Jose, and P.D. Gay. Affective feedback: an investigation into the ole of emotions in the infomation seeking pocess. In Poceedings of the ACM SIGIR Confeence on Reseach and Development in Infomation Retieval, pages 395 402. ACM, 2008. [2] F.J. Beniei and J.S. Gillis. Judging appot: Employing bunswik s lens model to study intepesonal sensitivity. In J.A. Hall and F.J. Beniei, editos, Intepesonal Sensitivity. Theoy and Measuement. Lawence Elbaum, 2001. [3] J.C. Biesanz and S.G. West. Pesonality coheence: Modeating self othe pofile ageement and pofile consensus. Jounal of Pesonality and Social Psychology, 79(3):425 437, 2000. [4] E. Bunswik. Peception and the epesentative design of psychological expeiments. Univesity of Califonia Pess, 1956. [5] A. Budick, J. Ducke, P. Lunenfelds, T. Pesne, and J. Schnapp. Digitial Humanities. MIT Pess, 2012. [6] D. Comaniciu and P. Mee. Mean shift: a obust appoach towad featue space analysis. IEEE Tansactions on Patten Analysis and Machine Intelligence, 24(5):603 619, 2002. [7] J.W. Condon and W.D. Cano. Infeed evaluation and the elation between attitude similaity and intepesonal attaction. Jounal of Pesonality and Social Psychology, 54(5):789, 1988. [8] R. Cowie. The good ou field can hope to do, the ham it should avoid. IEEE Tansactions on Affective Computing (to appea), 2013. [9] R. Datta, D. Joshi, J. Li, and J. Wang. Studying aesthetics in photogaphic images using a computational appoach. In Poceedings of the Euopean Confeence on Compute Vision, volume 3953 of Lectue Notes in Compute Science, pages 288 301. Spinge Velag, 2006. [10] J. Elste. Explaining Social Behavio. Cambidge Univesity Pess, 1997. [11] D.C. Evans, S. D. Gosling, and A. Caoll. What elements of an online social netwoking pofile pedict taget-ate ageement in pesonality impessions. In Poceedings of the Intenational Confeence on Weblogs and Social Media, pages 45 50, 2008. [12] P. F. Felzenszwalb, R. B. Gishick, and D. McAlleste. Disciminatively tained defomable pat models, elease 4. http://www.cs.bown.edu/ pff/latent-elease4/, 2010. [13] R. Fidel. Human Infomation Inteaction. MIT Pess, 2012. [14] S. Fitzgeald, D.C. Evans, and R.K. Geen. Is you pofile pictue woth 1000 wods? Photo chaacteistics associated with pesonality impession ageement. In Poceedings of AAAI Intenational Confeence on Weblogs and Social Media, 2009. [15] Wold Economic Foum. Pesonal data: the emegence of a new asset class. Technical epot, Wold Economic Foum, 2011. [16] C.M. Geogescu. Synegism in low level vision. In Poceedings of the Intenational Confeence on Patten Recognition, pages 150 155, 2002. [17] D. Johnson and J. Gadne. Pesonality, motivation and video games. In Poceedings of the Confeence of the Compute-Human Inteaction Special Inteest Goup of Austalia on Compute-Human Inteaction, pages 276 279, 2010. [18] N. Jojic and A. Peina. Multidimensional counting gids: Infeing wod ode fom disodeed bags of wods. In Poceedings of Uncetainty in Atificial Intelligence, pages 547 556, 2011. [19] C.M. Judd, L. James-Hawkins, V. Yzebyt, and Y. Kashima. Fundamental dimensions of social judgment: Undestanding the elations bteween judgments of competence and wamth. Jounal of Pesonality and Social Psychology, 89(6):899 913, 2005. [20] A.M. Kaplan and M. Haenlein. Uses of the wold, unite! The challenges and oppotunities of social media. Business Hoizons, 53(1):59 68, 2010. [21] M. Kosinski, D. Stillwell, and T. Gaepel. Pivate taits and attibutes ae pedictable fom digital ecods of human behavio. Poceeedings of the National Academy of Sciences, 110(15):5802 5805, 2013. [22] K. Kippendoff. Reliability in content analysis.
Human Communication Reseach, 30(3):411 433, 2004. [23] Z. Kunda. Social cognition: Making sense of people. The MIT Pess, 1999. [24] J. Leskovec, L. Adamic, and B. Hubeman. The dynamics of vial maketing. ACM Tansactions on the Web, 1(1):5, 2007. [25] M.S. Lew, N. Sebe, D. Chabane, and R. Jain. Content-based multimedia infomation etieval: State of the at and challenges. ACM Tansactions on Multimedia Computing, Communications, and Applications, 2(1):1 19, 2006. [26] P. Lovato, A. Peina, N. Sebe, O. Zandoná, A. Montagnini, M. Bicego, and M. Cistani. Tell me what you like and I ll tell you what you ae: disciminating visual pefeences on Flick data. In K.M. Lee, Y. Matsushita, J.M. Rehg, and Z. Hu, editos, Poceedings of the Asian Confeence on Compute Vision, volume Lectue Notes in Compute Science 7724. Spinge Velag, 2012. [27] J. Machajdik and A. Hanbuy. Affective image classification using featues inspied by psychology and at theoy. In Poceedings of the ACM Intenational Confeence on Multimedia, pages 83 92, 2010. [28] F. Maiesse, M. A. Walke, M. R. Mehl, and R. K. Mooe. Using linguistic cues fo the automatic ecognition of pesonality in convesation and text. Jounal of Atificial Intelligence Reseach, 30:457 500, 2007. [29] G. Machionini. Human infomation inteaction eseach and development. Libay & Infomation Science Reseach, 30(3):165 174, 2008. [30] K.V. Madia and P.E. Jupp. Diectional Statistics. Wiley Seies in Pobability and Statistics. Wiley, 2009. [31] M. Minelli, M. Chambes, and A. Dhiaj. Big Data, Big Analytics. Wiley, 2013. [32] C. Nass and S. Bave. Wied fo speech: How voice activates and advances the Human-Compute elationship. The MIT Pess, 2005. [33] R.E. Nisbett and T.D. Wilson. The halo effect: Evidence fo unconscious alteation of judgments. Jounal of Pesonality and Social Psychology, 35(4):250 256, 1977. [34] F.J. Olhost. Big Data Analytics. Wiley, 2013. [35] A. Oliva and A. Toalba. Modeling the shape of the scene: A holistic epesentation of the spatial envelope. Intenational Jounal of Compute Vision, 42(3):145 175, 2001. [36] M. Pantic and A. Vinciaelli. Implicit Human-Centeed Tagging. IEEE Signal Pocessing Magazine, 26(6):173 180, 2009. [37] A. Peina and N. Jojic. Image analysis by counting on a gid. In Poceedings of the Intenational Confeence on Compute Vision and Patten Recognition, pages 1985 1992, 2011. [38] A. Peteson Bishop, N.A. van House, and B.P. Buttenfields, editos. Digital Libay Use. MIT Pess, 2003. [39] B. Rammstedt and O. P. John. Measuing pesonality in one minute o less: A 10-item shot vesion of the Big Five Inventoy in English and Geman. in Jounal of Reseach in Pesonality, 41:203 212, 2007. [40] S. Ray. Multiple instance egession. In Poceedings of the Intenational Confeence on Machine Leaning, pages 425 432, 2001. [41] D. Rosenbeg. Data befoe the Fact. In L. Gitelman, edito, Raw data is an oxymoon, pages 15 40. MIT Pess, 2013. [42] G. Saucie and L.R. Goldbeg. The language of pesonality: Lexical pespectives on the five-facto model. In J.S. Wiggins, edito, The Five-Facto Model of Pesonality. 1996. [43] K.R. Schee. Pesonality makes in speech. In Social makes in speech, pages 147 209. Cambidge Univesity Pess, Cambidge, 1979. [44] K. Sohn, D.Y. Jung, H. Lee, and A.O. Heo. Efficient leaning of spase, distibuted, convolutional featue epesentations fo object ecognition. In IEEE Intenational Confeence on Compute Vision, pages 2643 2650, 2011. [45] J. Sule. The psychotheapeutics of online photoshaing. Intenational Jounal of Applied Psychoanalytic Studies, 6(4):339 344, 2009. [46] H. Tamua, S. Moi, and T. Yamawaki. Textue featues coesponding to visual peception. IEEE Tansactions on Systems, Man and Cybenetics, 8(6), 1978. [47] R. Tibshiani. Regession shinkage and selection via the lasso. Jounal of the Royal Statistical Society, Seies B, 58:267 288, 1994. [48] J.S. Uleman, L.S. Newman, and G.B. Moskowitz. People as flexible intepetes: Evidence and issues fom spontaneous tait infeence. In M.P. Zanna, edito, Advances in Expeimental Social Psychology, volume 28, pages 211 279. Elsevie, 1996. [49] J.S. Uleman, S.A. Saibay, and C.M. Gonzalez. Spontaneous infeences, implicit impessions, and implicit theoies. Annual Reviews of Psychology, 59:329 360, 2008. [50] P. Valdez and A. Mehabian. Effects of colo on emotions. Jounal of Expeimental Psychology Geneal, 123(4):394 409, 1994. [51] P.A. Viola and M.J. Jones. Robust eal-time face detection. Intenational Jounal of Compute Vision, 57(2):137 154, 2004. [52] J. Weise. Phototheapy techniques: Exploing the secets of pesonal snapshots and family albums. Jossey-Bass San Fancisco, 1993. [53] C.Y. Yaakub, N. Sulaiman, and C.W. Kim. A study on pesonality identification using game based theoy. In Poceedings of the Intenational Confeence on Compute Technology and Development, pages 732 734, 2010. [54] V. Yanulevskaya, J. Uijlings, E. Buni, A. Satoi, E. Zamboni, F. Bacci, D. Melche, and N. Sebe. In the eye of the beholde: employing statistical analysis and eye tacking fo analyzing abstact paintings. In Poceedings of the ACM intenational confeence on Multimedia, pages 349 358, 2012. [55] N. Yee, N. Ducheneaut, L. Nelson, and P. Likaish. Intoveted elves & conscientious gnomes: The