Are French Really That Different? Recognizing Europeans from Faces Using Data-Driven Learning

Are French Really That Different? Recognizing Europeans from Faces Using Data-Driven Learning Viet-Duy Nguyen *, Minh Tran * and Jiebo Luo Department of Computer Science, University of Rochester, Rochester NY 14627, USA Email: {vnguy14, mtran14}@u.rochester.edu, {jluo}@cs.rochester.edu Abstract Travel agents and retailers are curious about where their customers come from, which would help them increase their sales and optimize their marketing strategy. In this study, we present a system to predict where people come from in the European region only using their faces. The countries that have been chosen for the study are Russia, Italy, Germany, Spain, and France, based on diversity and representativeness. These countries have been well known for their economy, population, and political impact. First, we implement different neural network classifiers on the dataset of people s faces that we have collected from Twitter. Next, we investigate in more detail 11 different facial features that may help differentiate ethnic groups representative of those five countries. Our system achieves an accuracy of over 50%, more than twice as good as that of humans. Furthermore, we uncover and interpret using genetic anthropological evidences the various differences and similarities between people s faces across geographical distances among different contingents. I. INTRODUCTION From the following five images, can you tell the nationality of each person, given the choices are French, German, Italian, Russian, and Spanish? The answer, in order, is Russian, German, French, Italian, and Spanish. In fact, it is difficult to tell where people come from due to the relative distances between European countries. France, Germany, Italy, Russia and Spain are among the largest economies and populated countries in Europe 1,2, so these five countries have been chosen for the study. In the first stage of the study, we implement two different state-of-the-art neural network architectures, fine-tuned VGG- Net 16 [1] and fine-tuned ResNet50 [2]. On our collected Twitter dataset, with ResNet50, we achieve a remarkable accuracy of 53.2%, with the chances being 20%. Moreover, a user experiment has been conducted on 72+ individuals 3 and the human performance recorded so far is 26.96%. In the second stage of the study, we intend to explore informative facial features that help distinguish among European people. We use the CelebA Dataset [3] to train our model 11 times for 11 chosen facial attributes. We then use the trained models to classify images from our collected Twitter dataset. We have obtained interesting insights that can be inferred from our classification results. For example, Italians wear eyeglasses on their social media images more frequently than others, * Both authors contributed equally to this work. 1 http://databank.worldbank.org/data/download/gdp.pdf 2 http://databank.worldbank.org/data/download/pop.pdf 3 The experiment is available at https://tinyurl.com/europefaces Fig. 1. It is challenging to tell the differences between German, Italian, Spanish, Russian, and French. while Russians rarely wear eyeglasses. Moreover, the analysis on cross-country gender differentials is also reported. Finally, we use the trained models in the first stage to classify people s faces from around the world. We choose as inputs to our classification: Turkey, Saudi Arabia and Japan for Asia; Egypt for Africa; Cuba and Peru for America. Literature in History and Genetic Anthropology has been used to provide explanation for the results we obtain; for example, Turkish people resemble Italians the most, and Japanese look like Russians the most among the five European countries. In summary, we make several contributions in this study. First, using machine learning techniques, we build a system for recognizing Europeans from faces, which is about twice as good as humans. Second, we present a novel way to explain ethnic groups that share similar Y-DNA haplogroup distribution and related history solely by the analysis on faces. II. RELATED WORKS The study is related to the topics of deep learning for face and facial attribute classification, as well as genetic anthropology. In the field of face classification by neural networks, there are successes in terms of accuracy [4] thanks to large datasets [5], [6], [7]. The problem of race classification based on face images has been successfully tackled with an accuracy up to 98% as summarized by Fu et al.[8]. A few architectures are proposed [1], [2], [9], [10], [11]. Deep learning has been applied to not only face classification but also facial attribute classification. [12] is one of the earliest works in the field. [3] has achieved a great performance by training two networks. The first is for face localization and the second is for attribute classification. Following that work, [13] works well on a similar dataset with a different approach: using OpenCV to locate faces and training neural networks for every attribute. Our work is inspired by [13] for both face and facial attribute classification. We use VGG-Net16 and ResNet50 for face classification. [13] investigates data from three East

Asian countries (Japan, China, and Korea) with an accuracy of 75.03% for face classification. Our work has expanded the number of countries from 3 Asian countries to 5 European countries with the expectation of a lower accuracy due to the increased complexity. Instead of the neural network used in [13] for facial attribute classification, we train ResNet50 for each facial attribute. In the field of genetic anthropology, there is a study on the distribution of Y-DNA haplogroups in the populations of Europe [14]. The study suggests that unlike the other four countries, Russia tends to have most R1a and N in its population (46% and 23%, respectively). Although France, Germany, Spain and Italy have R1b as the haplogroup with the highest frequency, the second highest frequency haplogroup in these countries is different. For Spain, E1b1b is the second highest frequency haplogroup, while it is J2 for Italy. Both France and Germany have I1 as their second highest frequency haplogroup, but the frequency of I1 in Germany is about twice that of France (16% vs. 8.5%). This is clear evidence that there must be some measurable differences in facial appearance between the people from these five countries, which makes it possible to predict the nationalities of these people with a reasonable accuracy. Besides Europe, there exists much research on the distribution of Y-DNA haplogroups of countries all over the world. [15], [16] and [17] report the percentages of Y-DNA haplogroups in Japan, Saudi Arabia and Turkey, respectively. By studying the changes among different haplogroups, it is possible to project a map of migration flows of people from different countries around the world to Europe, or vice versa. III. DATASET This study uses two main datasets: Twitter images and CelebA dataset. The Twitter dataset contains labeled face images of 11 countries (France, Germany, Spain, Italy, Russia, Japan, Saudi Arabia, Turkey, Cuba, Peru and Egypt). The first five countries images are used to train the neural networks, while the others are used as the input of classification. The CelebA dataset is used to train the classification models for 11 facial features. A. Twitter Images The Twitter images are collected in a similar way for both Part 1 and Part 3 of the study. Part 1 of the study uses images of French, German, Spanish, Italian and Russian people. Part 3 of the study uses images of Saudi Arabian, Turkish, Egyptian, Japanese, Cuban and Peruvian people. Though the data is not ultimately clean, we tried our best to reduce cases that somebody of mixed descent by carefully choosing local celebrities or TV shows to retrieve data. First, we pick an influential Twitter account for each country and collect the followers of these accounts: French centrist politician Francois Bayrou for France, German reality talent show Deutschland sucht den Superstar for Germany, Spanish politician Mariano Rajoy for Spain, Italian Prime Minister Paolo Gentiloni for Italy, Russian Ice Hockey Federation Fig. 2. Resnet 50 Training loss. The graph indicates that the convergence is achieved. Twitter page for Russia, Cuban blogger Yoani Mara Snchez Cordero for Cuba, the Official Twitter Account of the Presidency of the Republic of Turkey for Turkey, President of Egypt Abdel Fattah el-sisi for Egypt, President of Peru Pedro Pablo Kuczynski for Peru and Japanese actor Tsuyoshi Muro for Japan. Since the initial dataset at this stage contains much noise, we restrict the list of followers based on the language they use on Twitter. We detect the language from the followers names and profile descriptions by using Google Language APIs [18] and from their Twitter language settings. Next, only images containing exactly one face are kept [19]. These images dimensions are resized to 224x224 pixels to fit the standard input for the neural network classification models [9]. Finally, we remove all images that have a size lower than 25kB to ensure high-quality images in our dataset. There are 30,806 French images, 34,265 German images, 31,929 Spanish images, 28,033 Italian images, and 32,365 Russian images; plus 15,049 Japanese images, 22,260 Saudi Arabian images, 44,480 Turkish images, 29,837 Cuban images, 41,945 Peruvian images and 42,857 Egyptian images. Our Twitter data collection method is not reliable to collect people s faces from English-speaking countries. The reason is that English is a widely used language in the British Commonwealth, which would lead to a problem of discerning where an English speaker is from. Therefore, it is not effective to apply the method above to collect the people s faces from countries such as the United States, Britain, or Australia. That is why British is excluded from this study (but can be added later with a better data collection method). B. CelebA Dataset For the second stage of the study, we use the CelebA Dataset [3]. The CelebA Dataset is a large-scale face attribute dataset with 10,177 identities and 202,599 images. Each image contains 40 annotated-binary face attributes, 11 out of which are chosen for this study. They are Eyeglasses, No beard, Chubby, Smile, Bangs, Rosy cheek, High cheekbones, Narrow

Fig. 3. On the left is the confusion matrix of ResNet50. On the right, the first two columns of the table are the result of facial features classification, which shows that our model gives a better result than LNet+ANet [3] on most of the 11 attributes. The second two columns are the probabilities of individual features for both genders. eyes, Pointy nose, Bushy, and Heavy makeup. The resulting models will be used to classify the facial attributes of people from the five countries. IV. METHOD In the first two stages of the study, we employ the cross validation method and divide our dataset into the ratio of 8 : 1 : 1 corresponding to training set : validation set : test set. A. Face Classification 1) Analysis on European Faces: Fine-tuned models often give a significantly higher accuracy than the original models [20]. With that notion, we use pre-trained VGG-Net 16 and ResNet50 on ImageNet [21] to train on our dataset. We reach an accuracy of 28.1% for fine-tuned VGG-Net 16 and a higher accuracy of 53.2% for fine-tuned ResNet50 after 120,000 iterations. Moreover, all of the models have been trained on the Caffe Framework and converge. 2) Analysis on Countries outside Europe: We apply our trained ResNet50 model not only in the prediction task but also in analyzing people s faces from places outside Europe. Using the trained model, we obtain the probabilities of the five classes (Spanish, Russian, German, Italian, and French) for every person s face picture from the six countries in Asia, Africa, and America. For each country outside Europe, we calculate the percentage of pictures that have been classified into a specific country among our five initial European countries (Figure 6). B. Facial Attribute Classification Given that ResNet50 gives us a high accuracy in classifying people s faces, we train it on CelebA Dataset 11 times for 11 different facial attributes. The second column of the table in Figure 3 is the result of our training on the facial attributes. Next, we implement facial attribute classification on our collected Twitter dataset using the model. However, we realize that there is an imbalance between the proportion of males and females in this study (80% vs. 20%). Therefore, we decide to perform a cross-country gender differential analysis. Some features have very low probabilities to appear on both the male and female (the forth and fifth columns of the table in Figure 3), which might make it difficult to achieve a good generalization of the whole population for the five countries. Therefore, we set the threshold to be 0.05, and the facial attributes that satisfy this criteria are Chubby, No beard, Eyeglasses and Smile. V. FACIAL ATTRIBUTES From the results of the facial attribute classification, we continue to extract further insights about the differences between males and females on their Twitter profiles across the five countries on each of the three facial attributes (Chubby, Eyeglasses, and Smile). We try our best to demystify the findings from the study, though not all features are easy to explain and find literature support. A. Chubby First, we observe that, for both sexes, France, Spain, and Italy are the top three countries with chubby people, followed by Germany and Russia (Figure 4). The order fits the Prevalence of Overweight for Adult by WHO in 2016 [22]. Furthermore, we can see that the percentage of chubby males is much higher than that of females in all five countries (Figure 4, Figure 3). This is an evidence that women appear or portray themselves to look less chubby on their Twitter profiles. B. Eyeglasses First, the distribution of the Eyeglasses attribute for both sexes in Figure 4 can be explained by the cultures of the five countries. France, Italy and Spain are well-known countries for

Fig. 4. Cross-country gender differentials on Smile, Eyeglasses, and Chubby. Red represents male, while Blue represents female. Fig. 5. Map of possible migration flow of populations around the world based on inferences of differences between Y-DNA distributions of countries. The map is designed by Bert Chen and posted at https://www.behance.net/gallery/31696657/infographics-y-dna-genetics their fashion industry, as reported in [23] as the three worldleading fashionable countries. Therefore, it is reasonable that citizens of these three countries wears more eyeglasses than Russian or German (countries that are not in top 10). Second, we have the same result as [13] that males tend to wear eyeglasses more than females for all five countries. The reason for this finding was suggested to be the difference between work-pressure of men and women. C. Smile The Spanish tend to smile the most, followed by Italian, French and German people (around the same level). Russians smile the least in their Twitter profile images. This finding matches the Happy Planet Index by New Economics Foundation (NEF) in 2016 [24]. Unlike previous attributes, the percentages of smiling females are always higher than those of males. Although there

is no concrete interpretation for this, it is likely that females prefer to look friendly while males may want to look cool on their Twitter profiles. VI. INTERCONTINENTAL FACIAL SIMILARITIES In the third stage, we use the ResNet50 model that we have trained in the first stage of the study. We have chosen different countries from Asia, America and Africa: Japan, Turkey, Saudi Arabia, Peru, Cuba, and Egypt. The data of the six countries from Asia, Africa, and America is used as the input to our model. Feeding the Twitter images that are prepared for Part 3 into the classification model of the five European countries, we obtain the following interesting results. A. Spanish look-alike From Figure 6, it is clear that Cuban and Peruvian people look Spanish, with more than 40% chance a Peruvian or Cuban person would be classified as a Spanish person. This could be explained due to the colonization by Spain in these two countries. Cuba was a Spanish colony for over four centuries (1492-1898), beginning with the arrival of Spanish explorer Christopher Columbus and ending with the withdrawal of Spain from the island due to the Spanish-American War. Peru was a Spanish colony for four decades (1532-1572). Therefore, both Cuba and Peru might share significant genetic similarities with Spain, which could be the reason why people from these two countries look Spanish. There are few Peruvians who have been classified as the other people, which can be explained by the various migrations into South America in the 18th-20th century. B. Italian look-alike There are two ways to explain why Saudi Arabian and Turkish people look similar to Italian people (Figure 6): Y- DNA Haplogroup Distributions and Historical Background. First, both Saudi Arabia and Turkey used to be colonized by the Roman Empire. In 106 CE, Roman Emperor Trajan took over the Arab Kingdom of the Nabataeans as the province of Arabia, which represented the Roman Empires easternmost frontier [25]. The Roman Empire also occupied Asia Minor, which makes up of the majority of Turkey, after Attalos III of Pergamon bequeathed his kingdom to Rome in 133 BCE [26]. These facts imply that there was a flow of people from Italia to both Turkey and Saudi Arabia. Moreover, the similarities between Italian and Turkish & Saudi Arabian people can be explained using the distribution of haplogroups in these countries. In Italy, the dominant Y-DNA groups are R1b, J2 and E1b1b, corresponding to 39%, 15.5% and 13.5% respectively [14]. In Saudi Arabia, the dominant haplogroups are E1b1a, E1b1b and J (including J1 and J2), which cover 7.6%, 7.6% and 58% of the population respectively [16]. In Turkey, the dominant haplogroups are R1b, E1b1b and J, corresponding to 16.1%, 11.3% and 33.3% of the population, respectively [17]. Because of the resemblance in the distribution of haplogroups between the three countries, people from these countries should look similar to each other. The world map in Figure 5 shows possible migration flows in the world based on the changes among different haplogroups in the world, which includes the flow from Italy to Saudi Arabia and Turkey (the blue dashed-line). C. Russian look-alike According to Figure 6, Japanese have the highest percentage of pictures classified as Russian. This matches the projected migration flow by Y-DNA haplogroup (Figure 5). The group of people in Northern Japan brought the C and D halopgroups to Southern and Middle Russia via the sea route. D. French look-alike The high percentage of Egyptian people being classified as French (31.07%) can be explained by the occupation of Egypt by France from 1798-1801. In this period, Napoleon Bonaparte, a French General, took control of Egypt [27]. This is evidence that there were groups of people from those two countries migrating between each other. VII. CONCLUSION In summary, we have achieved an accuracy of 53.2% for classifying face images of French, German, Italian, Russian and Spanish people. A approach to hopefully further improving the accuracy is to use the fine-tuned DenseNet [28], which is a more up-to-date neural network architecture. Based on our exploration of the 11 different facial attributes, we have found the following interesting results about the people from the five countries: 1) For both sexes, France, Spain, and Italy are the top three countries that have chubby people, followed by Germany and Russia. 2) People from Italy, France and Spain tend to wear more eyeglasses in their Twitter profiles than people from Russia and Germany. 3) The Spanish tend to smile the most, followed by Italian, French and German people. The Russians smile the least in their Twitter profile images. 4) Men in all the five countries smile less than women in their Twitter profile images. 5) Men in all the five countries tend to wear eyeglasses more than women. Finally, we also study other countries outside the European region to see if they might share facial similarities using machine learning techniques. There are 6 countries outside Europe (Japan, Saudi Arabia, Turkey, Cuba, Peru, Egypt) which have been collected from Twitter for this stage of study. These images are processed using our trained ResNet50 model in the first stage. We discover interesting insight that can be explained by the historical background and Y-DNA halopgroups of the six countries compared to the five European countries (Germany, France, Italy, Russia, and Spain). ACKNOWLEDGMENTS We thank the support of New York State through the Goergen Institute for Data Science, and our corporate sponsor Cheetah Mobile.

Fig. 6. Pie Chart of the percentages of classes (French, German, Russian, Italian, and Spanish) present in each country outside Europe. REFERENCES [1] K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015. [2] K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition. International Conference on Computer Vision and Pattern Recognition, 2016. [3] Z. Liu, P. Luo, X. Wang, and X. Tang, Deep learning face attributes in the wild. International Conference on Computer Vision, 2015. [4] Y. Wang, Y. Li, and J. Luo., Deciphering the 2016 u.s. presidential campaign in the twitter sphere: A comparison of the trumpists and clintonists. Tenth International AAAI Conference on Web and Social Media, 2016. [5] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, Labeled faces in the wild: A database for studying face recognition in unconstrained environments, Technical report, University of Massachusetts, 2007. [6] P. J. Phillips, H. Wechslerb, J. Huangb, and P. J. Raussa, The feret database and evaluation procedure for face-recognition algorithms. Image and Vision Computing, pp. 295 306, 1998. [7] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao, Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. European Conference on Computer Vision, 2016. [8] S. Fu, H. He, and Z.-G. Hou, Learning race from face: A survey, IEEE transactions on pattern analysis and machine intelligence, vol. 36, no. 12, pp. 2483 2509, 2014. [9] A. Krizhevsky, I. Sutskever, and G. E. Hinton., Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012. [10] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, Imagenet classification with deep convolutional neural networks. In International Conference on Learning Representations, 2015. [11] J. G. Zilly, R. K. Srivastava, J. Koutnik, and J. Schmidhuber, Recurrent highway networks. arxiv:1607.03474, 2016. [12] N. Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdev., Panda: Pose aligned networks for deep attribute modeling. International Conference on Computer Vision and Pattern Recognition, 2014. [13] Y. Wang, H. Liao, Y. Feng, and X. Xu, Do they all look the same? deciphering chinese, japanese and koreans by fine-grained deep learning, arxiv:1610.01854, 2016. [14] Eupedia.com, Distribution of european y-chromosome dna (y-dna) haplogroups by country in percentage, 2017. [15] M. F. Hammer, T. M. Karafet, H. Park, K. Omoto, S. Harihara, M. Stoneking, and S. Horai, Dual origins of the japanese: common ground for hunter-gatherer and farmer y chromosomes, Journal of Human Genetics Volume 51, 2006. [16] K. K, H. Ali, G. A. M, L. J. M, C. V. M, and U. P. A, Saudi arabian y- chromosome diversity and its relationship with nearby regions, National Center for Biotechnology Information, 2009. [17] C. Cengiz, K. Roy, K. Toomas, K. Ersi, A. Sevil, C. G. L., L. A. S., and R. C. C., Excavating y-chromosome haplotype strata in anatolia, 114:127.https://doi.org/10.1007/s00439-003-1031-4, 2004. [18] N. Shuyo, Language detection library for java. http://code.google.com/p/language-detection, 2010. [19] A. Geitgey, Modern Face Recognition with Deep Learning. Medium Corporation, 2016. [20] Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature, 2015. [21] J. Deng, W. Dong, R. Socher, K. Li, L.-J.and Li, and Fei-Fei, Imagenet: A large-scale hierarchical image database. Proceedings of the International Conference on Computer Vision and Pattern Recognition, 2009. [22] Overweight and obesity, World Health Organization, 2017. [23] R. Dicker, 10 most fashionable countries, US News, 2016. [24] K. Jeffrey, H. Wheatley, and S. Abdallah, The happy planet index, New Economics Foundation, 2016. [25] J. Dombrowski, Arabia, roman province, The Encyclopedia of Ancient History, 2012. [26] S. Mitchell, Asia, roman province of, The Encyclopedia of Ancient History, 2012. [27] C. Cherfils, Bonaparte et l Islam d apres les documents francais & arabes. Pedone, 1914. [28] G. Huang, Z. Liu, K. Q. Weinberger, and L. V. D. Maaten, Densely connected convolutional networks, arxiv:1608.06993, 2016.