Visual semantics: image elements. Symbols Objects People Poses

Transcription

1 Visible Partisanship Polmeth XXXIII, Rice University, July 22, 2016 Convolutional Neural Networks for the Analysis of Political Images L. Jason Anastasopoulos (University of Georgia, Public Admin + Policy, Political Science, Georgia Informatics Institute) Dhruvil Badani (UC Berkeley, EECS) Crystal Lee (UC Berkeley, EECS) Shiry Ginosar (UC Berkeley, EECS)

2 Outline Background Image experiment - the people you pose with how race and gender of people politicians pose with affect perceptions. Race classifier for images using convolutional neural networks. Analysis of race in US House of Representative Facebook profile photos.

3 Visual semantics: image elements Symbols Objects People Poses

4 Images convey political meaning: symbols

5 Images convey political meaning: objects Source - usa4palin.com: Still from Sarah Palin s Amazing America

6 Images convey political meaning: people Source - haaretz.com: Netanyahu (left), Obama (middle), Abbas (right)

7 Images convey political meaning: poses Source - google.com: Image search for John Boehner

8 Political functions of images For politicians Signaling Partisanship/ideology. Policy positions. Homestyle (Fenno 1978) Qualification competence. Identification I am one of you. Empathy I care about your needs.

9 Political functions of images For politicians Signaling Partisanship/ideology. Policy positions. Homestyle (Fenno 1978) Qualification competence. Identification I am one of you. Empathy I care about your needs.

10 Political functions of images For news media Issue framing. Persuasion.

11 Hardware and software limitations Hardware Even small images are big data. One 200 x 200 image = 3 200x200 matrices or 1 vector of length 120,000.

12 Hardware and software limitations Software High dimensional statistical theory developed more recently. n + Asymptotics deals with properties of estimators as parameters, p. arge (possibly In modern machine learning applications, (n, p) + with a fixed number of

13 Image analysis renaissance in social science Hardware: Powerful CPUs and now GPUs in desktop computers (thanks gamers!)

14 Image analysis renaissance in social science Software Statistical theory for computing in high dimensions. Advances in numerical computing. Deep-learning frameworks: Torch, Tensorflow, Theano, Caffe.

15 Signaling and image features Symbols Objects People Poses

16 Signaling and image features Symbols Objects People Poses

17 Questions How do the group characteristics (gender, race, age, etc.) of people that Members of Congress pose with affect how they are perceived? Social media homestyle Is there evidence that Members of Congress use social media images to signal identification and empathy with constituents using group characteristics?

18 Questions How do the group characteristics (gender, race, age, etc.) of people that Members of Congress pose with affect how they are perceived? Social media homestyle Is there evidence that Members of Congress use social media images to signal identification and empathy with constituents using group characteristics?

19 The people you pose with experiment: Lou Barletta Lou Barletta (R-PA, 11) chosen for initial experiment because of relative obscurity and similar pictures with different groups of people. MTurk respondents randomly assigned one of 7 images with Barletta Alone Barletta by himself. Woman Barletta with a woman. Man Barletta with a white man. Black Barletta with African-American men. Asked a series of questions based only on the image.

20 Image treatments Alone Man Woman Af. American

21 What is your best guess of the political party that this politician belongs to? Beliefs about Barletta s party ID vary significantly by image shown. Alone: 39% guessed Democrat 61% guessed Republican Black: 58% guessed Democrat 42% guessed Republican Man: 42% guessed Democrat 58% guessed Republican Woman: 43% guessed Democrat 57% guessed Republican

22 What is your best guess of this politician s ideological orientation? Average by treatment groups Alone: Moderate. Black: Liberal. Man: Moderate. Woman: Moderate.

23 Does this politician seem honest and trustworthy? Perceived to be more trustworthy when pictured next to a woman.

24 Does the politician seem like a strong and decisive leader? Perceived to be a stronger leader when pictured next to a woman.

25 Does the politician seem knowledgeable about the issues? Perceived to be less knowledgeable when pictured next to an older white man.

26 Does the politician seem like someone who shares my values? (non-white respondents) Perceived by non-white respondents to share their values when pictured next to African-American men.

27 Barletta experiment conclusions Opinion of Barletta affected by group identity of individuals included in images. Race affected beliefs about partisanship/ideology and implied shared values. Gender affected beliefs in trustworthiness, honesty and decisiveness. Survey experiment expanding goal is to test which aspects of photos most strongly tied to perceptions of candidate ideology and party.

28 Questions How do the group characteristics (gender, race, age, etc.) of people that Members of Congress pose with affect how they are perceived? Social media homestyle Is there evidence that Members of Congress use social media images to signal identification and empathy with constituents using group characteristics?

29 Questions How do the group characteristics (gender, race, age, etc.) of people that Members of Congress pose with affect how they are perceived? Social media homestyle Is there evidence that Members of Congress use social media images to signal identification and empathy with constituents using group characteristics?

30 Data 300,000+ Facebook images with text posts for accounts of: 300 US House members. 56 US Senate members.

31 Goals and Methods Identify race of individuals pictured in Facebook profiles of House Members. Viola-Jones Algorithm Train a convolutional neural network race classifier. Explore how distribution of racial groups in photos compare to congressional district demographics, partisanship and ideology. Compare Facebook profile demographics with district demographics, party id and DW-Nominate scores.

32 Results Democrats and Republicans in the US House of Representatives have very different social media styles. Evidence that Democrats use Facebook images to elicit racial identification and empathy among constituents.

33 How you see an image Image as data. Eg 620x412 pixel image. You see: {Donald Trump, blue tie, black suit, blue background, anger}

34 How a computer sees an image x412 pixel image. 3 Channels: Red, Green, Blue 3 640x412 matrices of pixel intensity values. -or- 3x640x412 = 791,040 x 1 vector

35 Human image classification is robust

36 Machine image classification is error prone... Image Credit: Andrej Karpathy

37 Theoretical means of image feature extraction limited mostly to faces... Viola-Jones Object Detection Framework

38 Data driven approach/supervised machine learning approach train models utilizing pixel intensity data... Collect labeled images. Train a machine learning classifier. Test classifier accuracy. CIFAR-10 library of 32x32 labels images benchmark performance.

39 One layer neural network Inputs multiplied by weights and added create hidden layer. Hidden layer passed through activation function multiplied by another set of weights to generate class probabilities/scores. X 1 X 2 X 3 Simplest model discussed by psychologist Rosenblatt (1958)

40 Neural network activation functions Most common activation functions are sigmoid and tangent. Optimizing predictions requires Choice of activation functions and; Choice of weights.

41 Backpropagation Rumelhart, Hinton and Williams (1986) Selection of weights involves: Forward pass - Calculation of loss function. Backward pass Use of chain rule and stochastic gradient descent to iteratively calculate new weights.

42 Convolutional neural networks for image feature classification Multi-layer neural network involving series of activation functions on chunks of pixel data.

43 Convolutional neural networks for image feature classification Equivalent of passing pixel data through a number of filters. Discover which filter is activated by which labeled image category. Output is highest probability category given filter responses Image credit: Andrej Karpathy

44 Convolutional neural network: building a race classifier Labeled image data 60,000 high school yearbook images, ,000 images sampled from Congressional Facebook dataset we collected. Categories: White, African-American, East Asian, Hispanic. 16-layer CNN model for large-scale image recognition from CNN Model Zoo by Simonyan and Zisserman (2015): and

45 Convolutional neural network: building a race classifier Step 1: Identify faces using Viola- Jones algorithm. Image on the right is a Facebook photo from Representative Tammy Duckworth s (D-IL) profile.

46 Convolutional neural network: building a race classifier Step 2: Label race of faces.

47 Convolutional neural network: building a race classifier Step 3: Train CNN on labeled data.

48 Convolutional neural network: building a race classifier Step 4: Test classifier accuracy Avg. cross-validated accuracy rates of 90% for whites, 85% for African- American, 75% for Asian, 65% for HIspanic.

49 Convolutional neural network: building a race classifier Step 5: Estimate race of individuals in Congressional Facebook image set using trained model.

50 Race and partisanship in House Facebook image posts (white House members) White Democrats post Facebook photos of African-Americans at 4x the rate of white Republicans Hispanics at 1.2x the rate of white Republicans. Asians at 2x the rate of white Republicans.

51 Race and partisanship in House Facebook image posts (white House members) Even conditional on relevant district demographics, state and region fixed effects, evidence of conscious efforts by partisans to include/exclude racial groups in Facebook image posts. Democrats (white): +6% more African-Americans in posts. Republicans (white): +6% more whites in posts.

52 Identification and empathy: district demographics and Facebook image posts Do MCs strategically post photos of racial groups to engender identification and empathy from constituents? Overall strong evidence that they do. Strong relationship between % of racial group in district and % of racial group posted in Facebook profiles.

53 Identification and empathy: district demographics and Facebook image posts by party Strategic use of race in image posts much more evident among Democrats than Republicans

54 Identification and empathy: district demographics and Facebook image posts by party Y = % white in Facebook profile photos White Democrats more race conscious when posting FB photos. After conditioning on state and region fixed effects and district demographics, Democrats Facebook photos more likely to reflect racial/ethnic mix of district.

55 Identification and empathy: district demographics and Facebook image posts by party Representation = % White in Facebook profile photos % White in Congressional District Whites over-represented in Facebook photos of white Democrats and Republicans

56 Identification and empathy: district demographics and Facebook image posts by party Representation = % Black in Facebook profile photos % Black in Congressional District African-Americans under-represented in Facebook photos of Republican MCs by an average of about 3.8%

57 Identification and empathy: district demographics and Facebook image posts by party Representation = % Hispanic in Facebook profile photos % Hispanic in Congressional District Hispanics under-represented In Facebook photos of both parties, more so among Democrats

58 Identification and empathy: district demographics and Facebook image posts by party Representation = % Asian in Facebook profile photos % Asian in Congressional District Asians under-represented in Facebook photos of white Democrats.

59 Discussion Modern computational methods allow for the large scale analysis of images. Here we build a race classifier for images using convolutional neural networks.

60 Discussion Characteristics of people that politicians pose with shape perceptions. Democrats and Republicans in the US House of Representatives have very different social media styles. Evidence that Democrats use Facebook images to elicit racial identification and empathy among constituents.