This is a repository copy of Measuring the effect of public health campaigns on Twitter: the case of World Autism Awareness Day. White Rose Research Online URL for this paper: http://eprints.whiterose.ac.uk/127215/ Version: Accepted Version Proceedings Paper: Ahmed, W., Bath, P.A. orcid.org/0000-0002-6310-7396, Sbaffi, L. et al. (1 more author) (2018) Measuring the effect of public health campaigns on Twitter: the case of World Autism Awareness Day. In: Transforming Digital Worlds. iconference 2018: Transforming Digital Worlds, 25-28 Mar 2018, Sheffield, UK. Lecture Notes in Computer Science. Springer Verlag, pp. 10-16. https://doi.org/10.1007/978-3-319-78105-1_2 Reuse Items deposited in White Rose Research Online are protected by copyright, with all rights reserved unless indicated otherwise. They may be downloaded and/or printed for private study, or other acts as permitted by national copyright laws. The publisher or other rights holders may allow further reproduction and re-use of the full text version. This is indicated by the licence information on the White Rose Research Online record for the item. Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing eprints@whiterose.ac.uk including the URL of the record and the reason for the withdrawal request. eprints@whiterose.ac.uk https://eprints.whiterose.ac.uk/
Measuring the effect of public health campaigns on Twitter: the case of World Autism Awareness Day [Author names removed for peer review] [Affiliations and Addresses removed for peer review] Abstract. Mass media campaigns are traditional methods of raising public awareness in order to reinforce positive behaviors and beliefs. However, social media platforms such as Twitter have the potential to offer an additional route into raising awareness of general and specific health conditions. The aim of this study was to investigate the extent to which a public health campaign, World Autism Awareness Day (WAAD), could increase Twitter activity and influence the average sentiment on Twitter, and to discover the types of information that was shared on the platform during a targeted awareness campaign. This study gathered over 2,315,283 tweets in a two-month period. Evidence suggests that the autism campaign, WAAD, was successful in raising awareness on Twitter, as an increase in both the volume of tweets and level of positive sentiment were observed during this time. In addition, a framework for assessing the success of health campaigns was developed. Further work is required on this topic to determine whether health campaigns have any long lasting impact on Twitter users. Keywords: Social Media, Health Campaigns, Twitter, Health Informatics, Autism 1 Introduction Millions of pounds are spent each year in order to encourage behavior change among the public, e.g., campaigns that encourage the public to eat healthily, undertake exercise, or cease smoking [1]. The purpose of mass media campaigns is to promote healthy behavior changes and to discourage unhealthy behaviors by raising awareness [1]. However, there has been limited empirical research to measure the extent to which health campaigns have the potential to raise awareness on social media (e.g., Twitter). Although there have been evaluations of traditional public health campaigns [2], the effectiveness of social media health campaign is an important issue that is currently under-researched. Understanding the effectiveness of public health campaigns on social media, such as Twitter, would enable policy makers and public health experts to develop more focused campaigns that could target specific messages and improve understanding of health issues. The main aim of this study was to assess the extent to which the public health campaign, World Autism Awareness Day (WAAD) influenced standard sentiment on Twitter, in order to determine whether the campaign was successful. In particular, the research objectives were:
To investigate the extent to which WAAD raised awareness on Twitter by investigating the volume of tweets that were sent and received using specific keywords, and to investigate the effect on positive or negative sentiment during WAAD. To better understand the different classes of WAAD tweets during April 2nd 2015 by training a machine learning classifier to categorize tweets on this day. To analyse the structure of the conversation during WAAD, by creating a network graph of tweets related to WAAD. A successful health campaign defined for the purposes of this study are those which: Raises awareness of a health disease and or condition, so that it makes members of the public conscious about the impact of a disease. In relation to Twitter we developed a framework for assessing the successfulness of a campaign. A successful campaign on Twitter should fulfill the following criteria: The volume of tweets should increase during the day of the campaign. The positive sentiment of tweets should increase during the campaign. When examining the structure of the network an isolates group should be identifiable which is important because it indicates a number of unique and unconnected users will be tweeting. When examining the content shared on the platform the majority of tweets should relate to the campaign. 2 Methodology This study employed a case study research design, which seeks to concentrate on one aspect in detail [3]. This study follows a pragmatic approach, and the research methodology consists of a mixed methods approach utilizing machine learning, network analysis, and time series analysis. Twenty-four hashtags and keywords were used to monitor discussions on Twitter which related to various aspects of Autism which were most popular at the time, and included the following: #measles, Measles, #measlesoutbreak, MeaslesOutbreak, #MMR, Vaccination, Vaccinate, Vaccines, #Vaccines, MMR Vaccine, Vaccination AND MeaslesOutbreak, Vaccinate AND Measles, #Vaccinedebate, #Vaccineswork, #cdcwhistleblower, #Vaccinateyourkids, #Vaccineinjury, #VaccinesSaveLives, #adhd, #aspergers, #autism, adhd, aspergers, and autism. Data were collected between March 19th 2015 and the 9th May 2015, which included the 2015 WAAD day (April 2nd); a total of 2,315,283 tweets were gathered. Once duplicates and non-english tweets were removed, 1,710,121 tweets remained. This final dataset contained data that was collected over 1,234 hours and which contained 637,969 unique Twitter users. The data collection on WAAD day started Thu Apr 02 00:00:00 +0000, 2015 and ended Thu Apr 02 23:59:59 +0000, 2015. Discover- Text [5], a cloud-based text analytics software was used to classify tweets by machine learning.the time series analysis program Mozdeh [4], which uses Twitter s Search API, was used to gather Twitter data. Mozdeh uses the SentiStrength algorithm. SentiStrengh, when tested on Myspace comments, was able to predict positive emotion with 60.6% accuracy [6].
3 2.1 User demographics Twitter provides geolocation data with each tweet; this is known as gold standard geolocation data, as location information from users Twitter biography may not be accurate. However, not all users enable this feature, thus not all tweets will contain geolocation data. In total 15,601(0.91%) out of 1,710,121 tweets contained valid latitude and longitude data. The majority of tweeters were based in the USA and the UK (see Figure 1). Figure 1. World map to show the geographic spread of tweets that contained geolocation information for the tweets collected in this dataset. 3 Results This section provides the results of the study. Figure 1 displays a time series graph of all tweets that were sent during the period, and Figure 2 shows the volume of tweets related to WAAD over time alongside the average positive and negative sentiment. It is important to note that, for the time series graph, some of the fluctuations are caused by the time of day, with peaks during the daylight hours and troughs during the night. This is because up to 60% of all tweets have English language settings, and this may indicate sleeping time for the majority of English speaking users [7]. Note that for this graph and others there is poor coverage during the initial phase of data collection due to Twitter s rate limiting. Rate limiting occurs when the number of requests to Twitter exceeds their maximum number of requests, and can also occur when there is unusually high traffic on the platform requests for data may be limited. Figure 2. Time series for all tweets from 2015.3.19.00.00 to 2015.5.9.09.0.
Figure 3. The average positive and negative sentiment strength for tweets containing all of the hashtags used to gather the data. Average tweet sentiment (1 to 5) +ve = red, -ve = black (0 = no posts), Grey/Pink: subjective posts only; Green = proportion of subjective tweets. As can be seen from Figure 2, the largest peak corresponded to World Autism Awareness day, when a total of up to 245,463 tweets (in all languages) were sent and received during April 2 nd 2015. This accounted for 14.4% of the total sample of tweets collected. Figure 3 shows the frequency of the sentiment ex-pressed by WAAD related tweets over time: positive sentiment is represented by a red line, and negative sentiment is represented by a black line. The tweets were consistently more positive than negative as positive sentiment lines (red) are higher than the negative sentiment lines (black). There is also an increase in the frequency of subjective posts. When examining exact duplicates it is possible to investigate the most frequently occurring tweets in this dataset. We found that that 8 out of the 10 most popular retweets were related to WAAD, therefore there was an observed increased interest in this during this time period. The manually-coded set of tweets was used to train a machine learning classifier using a Naïve Bayes algorithm. The entire dataset of tweets was then classified: 68,547 (81%) of tweets were classified as related to WAAD and 16,455 (19%) of tweets were not related to WAAD. In order to undertake additional analyses, a network graph was developed using NodeXL (Social Media Research Foundation, n.d.). The resulting graph is shown in Figure 4. Group 1 Figure 4. Network analysis for tweets send and received on April 2nd, 2015 related to the keywords used to retrieve data
5 The network graph shows how different structures were formed during WAAD on April 2 nd 2015. The network graph demonstrates (in group 1) that more than half of the tweets do not contain an @ sign in their tweet, that is to say users do not mention other Twitter users. This can be inferred from G1 (group 1), which lists users by themselves, i.e., tweeting using one of the keywords without mentioning other users. It can also be inferred from the network graph that there are many disconnected participants, and these isolated cases are on the left hand side of the network graph within G1 (group 1). Therefore, (taken with the findings from sections 3.2 to 3.4) the majority of tweets on Twitter over this time period corresponded to the occurrence of WAAD, and there are large fragmented Twitter populations that are tweeting about WAAD, but not to each other. A network graph without an isolates group may indicate that very few users are conversing about the topic of interest. 4 Discussion The time series and sentiment analysis results showed that an awareness of WAAD was raised, illustrated by the vast peak in tweets on this date. The results from the machine learning classification demonstrated that the majority of content across this time period were directly related to WAAD. The network analysis demonstrated that a large part of the conversation was driven by WAAD tweets due to the large isolated group. Thus, by taking into account the four separate analyses above, the categories for a successful health campaign outlined earlier in the study were fulfilled, as shown in Table 1 below: Table 1. Criteria for a successful campaigns in relation to WAAD Criteria World Autism Awareness Day C.1 Volume of tweets should increase during the day of the campaign C.2 Positive sentiment of tweets should increase during the campaign C.3 An isolates group should be identifiable in the network graph C.4 When examining the content shared on the platform the majority of tweets should relate to the campaign. Volume of tweets increased as demonstrated by the results of the time series graph (figure 2). Positive sentiment of tweets increased during the campaign as demonstrated by the sentiment analysis graph (figure 3). An isolates group was formed during the campaign (figure 6). The majority of tweets (57%) related to WAAD. The implications of these results are that campaign managers could seek to promote the creation of news stories which are shareable. Moreover, for post-campaign monitoring examining the number of unique users in the form of isolates would provide an indication of the number of unique users engaging with a health campaign. Therefore, this study provides evidence to suggest that the WAAD campaign successfully raised
awareness on Twitter. It is possible that all campaigns may look like this, therefore future research could seek to compare the success of different campaigns. Moreover, future research could seek to examine other aspects which correlate with awareness, such as donation amounts, which could form a further measure of success. Moreover, additional data sources such as Google Trends could be utilised in order to strengthen future work. A more in-depth study could use the Firehose API (all available tweets); however, it may be that the tweets gathered for this study are upwards of 70% [8]. 5 Conclusion This initial study found some evidence to suggest that the autism campaign, WAAD, was successful in raising awareness on Twitter, in that, across this time period, there was an increase in both the volume of tweets, and their positive sentiment. However, we do not make any conclusions on whether the increased awareness on Twitter had an impact on individuals. Instead, we provide evidence that the WAAD campaign did appear to increase awareness on Twitter. Further work is required which examines the longevity and also compares a number of health campaigns in order to reach such a conclusion. Further work will seek to build a custom sentiment algorithm to classify tweets, and to utilise multiple coders to train the classifier. References [1] Randolph, W., & Viswanath, K. (2004). Lessons learned from public health mass media campaigns: marketing health in a crowded media world. Annual Review of Public Health, 25, 419 437. doi:10.1146/annurev.publhealth.25.101802.123046 [2] Hornik, R. (2002). Public health communication evidence for behavior change. Mahwah, N.J: Lawrence Erlbaum Associates. [3] Bryman, A. (2004) Social Research Methods (2nd edition). Oxford: Oxford University Press. [4] Mozdeh (n.d.). Mozdeh Twitter Time Series Analysis. [Online] Available at: http://mozdeh.wlv.ac.uk/ [Last accessed 19 May. 2015]. [5] Discovertext. (n.d.).discovertext. [Online] Available at: https://www.discovertext.com/ [Last accessed 12/02/2015]. [6] Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., & Kappas, A. (2010). Sentiment strength detection in short informal text. Journal of the Association for Information Science and Technology, 61(12), 2544-2558. [7] Gerlitz, C., & Rieder, B. (2013). Mining one percent of Twitter: Collections, baselines, sampling. M/C Journal, 16(2). [8] Social Media Research Foundation. (n.d.). NodeXL. Retrieved from http://www.smrfoundation.org/nodexl/ [Last accessed 28/05/15].