Event Summarization using Tweets

Transcription

1 Event Summarzaton usng Tweets ABSTRACT Deepaan Chakrabart Yahoo! Research 701 1st Avenue Sunnvale, CA Twtter has become exceedngl popular, wth mllons of tweets beng posted ever da on a wde varet of topcs. Leadng search engnes have started routnel dsplang relevant tweets n response to user search queres. Recent research has shown that a consderable fracton of these tweets are about events, and the detecton of novel events n the tweet-stream has attracted a lot of nterest. However, ver lttle research has focused on dsplang ths real-tme nformaton about events. For nstance, the leadng search engnes smpl dspla all tweets matchng the queres n reverse chronologcal order. In ths paper we argue that for some recurrng events, such as sports, t s better to use more sophstcated technques to summarze the relevant tweets. We formalze the problem of summarzng event-tweets and gve a soluton based on learnng the underlng hdden state representaton of the event va Hdden Markov Models. In addton, through extensve experments on real-world data we show that our model sgnfcantl outperforms some ntutve and compettve baselnes. 1. INTRODUCTION A ke component of real-tme search s the avalablt of realtme nformaton. Such nformaton has recentl prolferated thanks to socal meda webstes lke Twtter and Facebook that enable ther users to dscuss, comment, and otherwse communcate contnuousl wth others n ther socal crcle. On Twtter, user compose and send short messages called tweets, puttng the medum to a wde arra of uses. Recent research has shown that one of the bg use-cases of Twtter s users reportng on events that the are experencng: Sakak et al [12] demonstrated that careful mnng of tweets can be used to detect events such as earthquakes. Snce then a lot of research works have focused on detecton of novel events on Twtter [10] and other socal meda webstes [2] such as Flckr and Youtube. However, the state of the art n real-tme usage of ths data s stll rather prmtve. In response to searches for ongong events, toda s major search engnes smpl fnd tweets that match the quer terms, and present the most recent ones. Ths approach has the advantage of leveragng exstng quer matchng technologes, and for smple one-shot events such as earthquakes t works well. However, for events that have structure or are long-runnng, and where users are lkel to want a summar of all occurrences so far, ths approach s often unsatsfactor. Consder, for nstance, a quer about an ongong game of Amercan Football (we wll use ths as our runnng example). Just Returnng the most recent tweets about the game s problematc for two reasons: (a) the most recent tweets could be repeatng the same nformaton about the event (sa, the most recent touchdown ), and (b) most users would be nterested n a summar of the occurrences n the game so far. Ths approach can clearl be Coprght 200X ACM...$ Kunal unera Yahoo! Research 701 1st Avenue Sunnvale, CA kpunera@ahoo-nc.com mproved upon usng nformaton about the structure of the event. In the case of Football, ths corresponds to knowledge about the game pla. Ths motvates our goal of summarzng long-runnng structurerch events. Ths ncel complements recent work on detectng events n the twtter stream [10]. We that a new event has been detected, our goal s to extract a few tweets that best descrbe the chan of nterestng occurrences n that event. In partcular, we focus on repeated events, such as sports, where dfferent games share a smlar underlng structure (e.g., touchdowns, nterceptons, fumbles, n Amercan football) though each ndvdual game s unque (dfferent plaers, dfferent sequences of touchdowns, fumbles, etc.). We want our approach to learn from prevous games n order to better summarze a recent (perhaps even an ongong) game; queres about ths game can then present a full summar of the ke occurrences n the game, as well as the latest updates. Our proposed soluton s a two-step process. We frst desgn a modfed Hdden Markov Model that can segment the event tmelne, dependng on both the burstness of the tweet-stream and the word dstrbuton used n tweets. Each such segment represents one dstnct sub-event, a semantcall dstnct porton of the full event. We then pck ke tweets to descrbe each segment judged to be nterestng enough, and combne them together to buld the summar. OUR CONTRIBUTIONS. We are, to the best of our knowledge, the frst to stud the problem of summarzng events on Twtter. To accomplsh ths, our algorthm surmounts several challenges. (1) Events are tpcall burst. Some tpes of sub-events generate far more tweets per unt tme than others. Our algorthm ensures that the summar does not contan tweets from perods of low actvt, et does not over-represent the burst perods ether. (2) Separate sub-events ma not be temporall far apart. Our algorthm surmounts ths problem b automatcall learnng language models for common tpes of sub-events. Ths allows t to separate dfferent tpes of sub-events even when the are temporall close. (3) revous nstances of smlar events are avalable. Our algorthm can buld offlne models for the dfferent sub-events usng tweets from prevous events we do not want to re-learn the characterstcs of a touchdown n ever match. (4) Tweets are nos. Tweets are ver short (less than 140 characters) and nos, rfe wth spellng mstakes. Our algorthm acheves robust results n the face of these ssues. (5) Strong emprcal results. We emprcall compare ths model aganst several baselnes to demonstrate ts accurac; n partcular, our algorthm acheves better recall over the set of mportant occurrences, and also gves a better sense of the context of these occurrences and the plaers assocated wth them. ORGANIZATION. In Secton 2, we present a detaled case-stud on the need for real-tme summarzaton of coverage of Amercan Football games on Twtter. Ths also provdes a concrete motvaton for our work. In Secton 3, we dscuss several baselnes and then present

2 Fgure 1: Volume of queres to the Google search engne contanng names of both teams nvolved n NFL games. We see marked spkes on da of the games. The -axs values onl makes sense relatvel. Fgure 2: The absolute number of tweets referrng to Football team names. There are marked spkes on the das that the teams compete n games. our proposed soluton based on a modfcaton of Hdden Markov Models. These are compared n Secton 4, where the accurac and utlt of our algorthm s demonstrated. We defer the dscusson of related work to Secton 5 as presentng the detals of our approach frst helps us better contrast t wth exstng works. Fnall, we conclude n Secton 6 lstng drectons for future research. 2. SORTS COVERAGE ON TWITTER We want to summarze events so that the summares can be ntellgentl consumed b downstream applcatons lke search engnes answerng user queres. But s there an actual demand for such summares? In partcular: (1) Are web users nterested n real-tme nformaton about these events? (2) Is such real-tme nformaton avalable? (3) Are exstng methods to explot ths real-tme nformaton suffcent? THE DEMAND FOR REAL-TIME COVERAGE OF SORTS EVENTS. For the frst queston, we consder the example of users lookng for nformaton on professonal Amercan Football games. In Fgure 1 we plot the number of occurrences of three bgrams n the search quer logs of the Google search engne. The bgrams jets steelers, jets patrots, and jets colts refer to the names of teams that plaed each other on three dfferent Sundas n Januar Even though Google Trends 2 does not reveal absolute numbers, the graphs clearl ndcate a large spke n user nterest concurrent wth the games; ths nterest also extends nto the next da (when users tpcall access news stores). There s a smlar surge n user nterest for soccer matches durng the World Cup Clearl, an automated method to answer user queres about events usng realtme nformaton would have sgnfcant utlt. THE SULY OF REAL-TIME INFORMATION ON SORTS EVENTS. 1 lottng the frequenc of occurrences of the names of both teams n a game focuses n on the nterest n the game as opposed to just general nformaton about the teams. 2 Fgure 3: The absolute number of tweets referrng to Football teams plotted over the duraton of the game. It s clear that plas of nterest are marked b spkes n tweet volume. The top frequent words durng some spkes are marked showng that words n the tweets descrbe the plas. To answer the second queston, we consder the problem of obtanng real-tme nformaton about these events. One readl avalable source s Twtter, and prevous research has shown that some events such as earthquakes can be successfull dentfed b mnng user tweets [12]. We want to take ths one step further and mne user tweets to construct real-tme summares of events. Here we agan consder the example of Amercan Football games. For the purposes of ths experment we dentf tweets referrng to teams b lookng for certan hash-tags 3. In partcular, n Fgure 2 we plot the number of occurrences of tweets contanng the hash-tags #jets, #steelers, #patrots, and #colts. From the Fgure, t s clear that the tweets referrng to the teams spke durng the tmes when the events take place. The three games mentoned n Fgure 1 show up clearl n the tweets n addton to other games the teams are nvolved n. In order to check whether the tweets that contan references to the team names are about the game at all we show n Fgure 3 the frequenc of such tweets for two games over the duraton of the games. It s clear that the ebb and flow of the tweets s correlated to the happenngs of the game. In addton to the frequenc we have tagged some of the frequenc spkes wth the common words n the tweets. As we can see the words n the tweets assocated wth the spkes do document the happenngs at these events. CHARACTERISTICS OF SORTS COVERAGE IN TWEETS. As we have seen tweets that refer to Football teams nvolved n games tend to provde detals about specfc plas of nterest. In partcular, we defne as a sub-event a dscrete chunk of the full event that can stand on ts own: e.g. touchdown, ntercepton, etc. n case of a Football game. Hence, our goal n ths work s to construct a event-summar b selectng tweets, from the set of relevant tweets, that descrbe these sub-events. Here we lst some characterstcs and ssues of ths data that our proposed approach wll deal wth. Frst, notce from Fgure 3 that sub-events are marked b ncreased frequenc of tweets. Moreover, t s the change n tweet-frequenc as opposed to absolute levels that demarcate sub-events. Second, boundares of sub-events also result n a change n vocabular of tweets. The vocabular specfc to a sub-event tends to come from two sources: one part of t s sport specfc comprsng words lke touchdown, ntercepton, etc., whle another part s game-specfc, such as plaer names. Fnall, the sport events we are consderng are repeated events sharng at least the sport-specfc vocabular and hence our approach should be able to learn from past games to do a better job of summarzng future ones. Some or all of these ssues also pertan to other events such musc festvals or award shows. 3 When Twtter users want to refer to enttes n ther tweets the often use hash-tags; sntactcall, these are words prefxed wth # embedded n the tweet

3 So the fnal queston becomes whether exstng methods address these ssues. At all major search engnes, the current methods for surfacng the content from twtter n response to search queres nvolve lttle more than returnng the matchng tweets n the reverse chronologcal order. Ths technque works well for man use-cases of twtter, but not for summarzng events, as t does not take nto account the man of the propertes of event coverage on twtter descrbed above. We evaluated ths technque as a baselne n our experments n Secton 4 but do not report the numbers as t proved to be extremel uncompettve. In the next secton we propose a seres of approaches that are desgned to explot the propertes of coverage of events on Twtter. 3. ALGORITHMS Our goal s to summarze tweets about events n real-tme, allowng for ntellgent ntegraton of tweets nto search results. In ths secton, we wll dscuss three algorthms to accomplsh ths. In order to ad understandng of the ssues nvolved and the desgn choces we make, we start b presentng the smplest approach and progressvel repar ts shortcomngs usng more sophstcated methods. Also, whle the algorthms are gven for events n general, we contnue to use the runnng example of sportng events to elucdate deas. 3.1 Baselne SUMMALLTEXT The straghtforward approach to summarzng tweets s to smpl consder each tweet as a document, and then appl a summarzaton method on ths corpus. In partcular, we assocate wth each tweet a vector of the TF-logIDF of ts consttuent words [1]. Defnng the dstance between two tweets to be the cosne dstance between them, we select those tweets whch are closest to all other tweets from the event. The algorthm s formall descrbed n Fgure 3.1. Algorthm 1 SUMMALLTEXT INUT: Tweet corpus Z, tweet word vocabular V, desred number of tweets n OUTUT: Set of ke tweets T for Z, w V do z (w) = tfdf(w,, Z) end for for Z do score() = j Z cosne(z, z j ) end for T = top n tweets wth maxmum score ISSUES. Whle ths algorthm has the advantage of smplct, t stll has several defects. Frst, O( Z 2 ) computatons are requred to compute the scores (though ths could be reduced n practce f prunng technques and approxmatons such as LSH are used). More mportantl, the result set T s lkel to be heavl based towards the most popular sub-event, to the complete excluson of other subevents. The usual method used n such cases s to sequentall add those tweets to T that (a) have hgh scores, whle (b) are dverse enough from the the tweets alread added to T. However, smpl usng a text-based dverst functon wll not help n our case, snce we want T to contans tweets for ever major sub-event, even f those sub-events are of the same class for example, n football, we want to get tweets for all touchdowns, even though the tweets descrbng each of these touchdowns are presumabl qute smlarl worded. In essence, a tme-based dverst functon s necessar. 3.2 Baselne SUMMTIMEINT In order to pck tweets from the entre duraton of the event, we need to combne summarzaton wth a segmentaton strateg. The smplest dea s to splt up the duraton nto equal-szed tme ntervals (sa, 2 mnutes for football), and then select the ke tweets from each nterval. However, clearl, not all ntervals wll contan useful subevents. We detect such ntervals b ther low tweet volume relatve to the average, and do not select an ke tweets from such ntervals. The algorthm s descrbed n Fgure 3.2. It has two extra parameters: (a) a segmentaton T S of the duraton of the event nto equaltme wndows, and (b) the mnmum actvt threshold l, whereb tme segments where tweet volume s less than l% of all tweets are gnored. Both the wndow length and the threshold l are to be pcked expermentall wth the goal of havng no more than one complete sub-event n each tme segment. Algorthm 2 SUMMTIMEINT INUT: Tweet corpus Z, tweet word vocabular V, desred number of tweets n, mnmum actvt threshold l, tme segments T S OUTUT: Set of ke tweets T T S = {s T S tweet volume n segment s > l% of all tweets} for each segment s T S do Z[s] = Z restrcted to tme s T s = SummAllT ext(z[s], V, n/ T S ) end for T = S T s ISSUES. Algorthm 3.2 ensures that the selected tweets are spread across the entre duraton of the event. Dverst wthn an gven tme segment now becomes less useful, because onl a few tweets are pcked from each segment. However, the segmentaton based on equal-szed tme wndows s far too smplstc, for three reasons dscussed below. Burstness of tweet volume: Tweets come n bursts, and the duratons of these bursts can var. If the event s splt nto constant-tme stages, one sngle long burst can be splt nto multple stages, and the ke tweets from each stage are lkel to be near-duplcates. Conversel, f each stage s too long, t mght cover several sub-events n addton to the burst sub-event; snce onl a few tweets can selected from each tme segment, some sub-events are lkel to be mssng from the fnal set of ke tweets. The problem remans even f the event s splt nto stages wth constant number of tweets n each stage. Clearl, none of these trval solutons s satsfactor. Multple sub-events n the same burst: Even f bursts n tweet volume can be detected accuratel, segmentng b burst volume can be msleadng. For nstance, f two sub-events occur close together n tme (e.g., an ntercepton followed soon after b a touchdown ) and both generate sgnfcant tweets, then ther respectve tweets mght get smeared nto one seemngl contnuous burst n tweet volume. However, a careful examnaton of the word dstrbuton of the tweets n ths burst would reveal the presence of the two dfferent sub-events. Thus, a good segmentaton should consder changes n language model along wth changes n tweet volume. Cold Start : At the begnnng of the game, when the tweet language models are unknown and the thresholds defnng burst behavor are unclear, the segmentaton algorthm could be ver naccurate. Ths s especall the case f the thresholds for bursts or changes n language model have to be learnt automatcall. Thus, there s a sgnfcant rsk of mssng mportant sub-events that happen earl. We need a segmentaton method that can overcome all these obstacles. A good segmentaton wll solate at most one sub-event n each tme segment, whch can then be used b the summarzer to output summares of each segment (or choose to gnore segments where lttle seems to be happenng). In the next secton, we propose a varant of the Hdden Markov Model (HMM) to solve ths problem.

4 3.3 Our Approach: SUMMHMM We have seen that there are two parts to event summarzaton: detectng stages or segments of an event, and summarzng the tweets n each stage. For nstance, perods of sgnfcant tweetng actvt (e.g, after a soccer goal or a football touchdown ) mght be nterleaved wth lulls n tweet volume, or the predomnant personalt beng dscussed n the tweet-stream of an event could change from one proper name to another, and so on. Accurate knowledge of the boundares between stages s crucal n fndng the best tweets about the most mportant sub-events. To segment an event, we turn to a model that has worked ver well for man such problems: the Hdden Markov Model (HMM). The HMM s able to learn dfferences n language models of subevents completel automatcall, so that the model parameters are tuned to the tpe of event. The parameters of the HMM are easl nterpretable as well. These advantages make t eas for the HMM to be appled to a wde varet of events, and make t our tool of choce for event segmentaton. However, the standard HMM requres some modfcatons to be applcable to our problem, so we frst present a short background on HMMs before dscussng our partcular HMM. BACKGROUND ON HMMS. The standard HMM posts the exstence of N states labeled S 1,..., S N, a set of observaton smbols v 1,..., v M, the probabltes b (k) of observng smbol k whle n state, the probabltes a j of transtonng from state to state j, and the ntal state dstrbuton π [11]. Startng from an ntal state, the HMM outputs a smbol pcked accordng to the smbol probabltes for that state, and then transtons to another state based on the transton probabltes (self-transtons are allowed). Now, gven several sequences of smbols, one ams to learn the smbol and transton probabltes of the HMM that best ft the observed sequences. In our case, each state could correspond to one class of sub-events (e.g., touchdown, ntercepton, fumble, etc.), and the smbols are the words used n tweets. Thus, our event HMM models each event as a sequence of states, wth tweets beng the observatons generated b the states. The varaton n smbol probabltes across dfferent states account for the dfferent language models used b the Twtter users to descrbe dfferent classes of sub-events, and the transtons between states models the chan of sub-events over tme that together make up an gven event Our Modfcatons We enhance the standard HMM wth several modfcatons that make the HMM more relevant to event segmentaton. OUTUTS ER TIME-STE:. One dfference between our HMM and the standard HMM s mmedatel clear the observaton from a gven state of our HMM conssts of all tweets for that tme perod (.e., a multset of smbols) nstead of just one smbol, as n the standard HMM. Ths requres a smple extenson of the standard model. DETECTING BURSTS IN TWEET VOLUME:. Another dfference s that the standard HMM does not account for dfferent rates of tweets over tme, snce t onl outputs one smbol per tme-step. Thus, t s unable to model bursts n tweet volume. Instead, we allow each state to have ts own tweet rate whch models the expected fracton of tweets n a game whch come from that state. Ths allows for dfferentaton between states on the bass of the burstness of the tweet-stream, whch complements the dfferentaton based on the language model of state-specfc smbol probabltes. COMBINING INFORMATION FROM MULTILE EVENTS:. Note that n order to learn the parameters of the HMM, we requre several observaton sequences generated b the HMM. In fact, f we buld an HMM for just the current event, usng onl the tweets seen untl now, then t wll be ver smlar to an algorthm that smpl detects change-ponts n a data stream. Whle change-pont detecton sstems are ver useful n practce, the suffer from the problem of cold-start: ever tme there s a change-pont, a new model of the data must be learnt, so there s a tme lag before the next change-pont can be detected. Also, snce the change-pont sstem can onl model the tweets t has seen so far, t can be slow to trgger when a new class of sub-event occurs, explanng awa the frst wave of tweets from the new sub-event as just the varablt of the current model. Clearl, modelng each event b tself has dsadvantages. Ths motvates learnng the HMM parameters b tranng on all avalable events of a certan tpe (e.g., all football games n a season). Snce all football games share the same classes of sub-events ( touchdown, ntercepton, etc.), combnng the data from multple events allows us to (a) better learn the HMM parameters, and (b) better detect state transtons n a new game, thus solvng the cold-start problem. However, ths also has the dsadvantage that the HMM has to account for tweet words that onl occur n some of the events, but not n others. The most common example of ths s proper names. For nstance, tweets about two dfferent football games wll almost never share plaer names or team names. However, such proper names could be ver mportant n dstngushng between states (e.g., certan plaers onl pla n defense or offense), and so gnorng names s not a soluton. To account for ths, we mantan three sets of smbol probabltes: (1) θ (s), whch s specfc to each state but s the same for all events, (2) θ (sg), whch s specfc to a partcular state for a partcular game, and (3) θ (bg), whch s a background dstrbuton of smbols over all states and games. Thus, θ (s) encapsulates dfferent classes of sub-events, whle θ (sg) captures proper names and other smbols that can var across games but stll gve some sgnal regardng the correct state; fnall, θ (bg) handles all nos or rrelevant tweets. The standard HMM uses onl θ (s). Ths dfferentaton of smbol probabltes across specfc events s another aspect of our HMM that dstngushes t from the standard HMM Mathematcal Formulaton As n the standard HMM, let S represent the set of states, V the set of observaton smbols (.e., the tweet vocabular), and a j the probablt of transtonng from state to state j. We shall use, j S to represent states, x, V to represent a smbol, g to represent one partcular event, and t to represent a tme-step. We assume that the fracton of tweet words n the corpus that are generated b state s gven b a Geometrc dstrbuton wth parameter κ. Let φ (s), φ (sg), and φ (bg) represent the probabltes that a smbol generated b state s pcked accordng to θ (s), θ (sg), or θ (bg) respectvel. Let θ (s), θ(sg), and θ(bg) represent the probablt of beng generated b the state-specfc dstrbuton for state, the dstrbuton for state and game g par, or the background dstrbuton respectvel. Together, Θ = {a, κ, φ (s), φ (sg), φ (bg), θ (s), θ(sg), θ(bg) } s the set of unknown parameters of the HMM. Let Z be the sequence of observatons from all the events n the corpus, X the sequence of states for each of those events, and W the sequence of smbol dstrbutons (θ (s), θ (sg), or θ (bg) ) from whch all the smbols n Z are generated. Thus, the par (X, W ) s the set of hdden varables. We also defne some auxlar varables. Let N (s), N (sg), and represent the number of nstances n whch s generated b N (bg) θ (s), θ (sg), and θ (bg) ; these are multnomal random varables whose values are known once X and W are fxed. Let N j represent the number of transtons from state to j; ths too can be computed gven X and W. Fnall, let N gt be the number of smbols n game g at tme-step t, and let N g = t Ngt; both of these can be computed

5 γ gt() = X j sum = θ (s) = X g,t ζ (sg) = X t ζ (bg) = X,g,t ξ gt(, j) φ(s) + θ (sg) φ(sg) + θ (bg) φ (bg) I(Z gt = )γ gt() θ(s) φ(s) sum I(Z gt = )γ gt() θ(sg) φ(sg) sum I(Z gt = )γ gt() θ(bg) φ (bg) sum Our modfed HMM takes multple events of the same tpe as nput, and learns the model parameters Θ that best ft the data. Gven Θ, the optmal wa to segment the events can be quckl found b the standard Vterb algorthm [11], whch we do not descrbe here. Each segment can then be summarzed, eldng the fnal set of top tweets for each event. Note that onl the computaton of Θ s tme-consumng, and ths can be done perodcall and offlne. Then, for a new event, the segments can be computed onlne, usng an old θ, as new tweets are generated. Fgure 3 gves the pseudo-code for our approach. θ (s) = θ (sg) = θ (bg) = φ (s) = φ (sg) = φ (bg) = a j = π = κ = x ζ(s) x ζ (sg) x ζ(sg) xg ζ (bg) x ζ(bg) x g,t ξgt(, j) g,t γgt() g γg1() gj γg1(j) g,t ζ(s) + g ζ(sg) + ζ(bg),g ζ(sg) + g ζ(sg) + ζ(bg) ζ(bg) + g ζ(sg) + ζ(bg) g,t γgt() γgt() (Ngt + 1) Table 1: One sngle teraton of the EM algorthm. from the data Z. Now, the complete data lkelhood s gven b: " Y,g " Y,,g (Z, X, W Θ) = (1 κ ) φ (s) N (s) Y,j N (s) θ (s) φ (sg) a Nj j! Y,g «1 # (sg) (bg) +N +N Ng κ π I(X g1=) N (sg) θ (sg) φ (bg)! θ (bg) N (bg) # where I(.) s the ndcator functon. Our goal s to fnd the best parameter settng Θ that maxmzes (Z Θ). We do ths b extendng the EM algorthm to our HMM. Defne ξ gt(, j) to be the probablt that there s a transton from state to state j at tme t n game g. The computaton of ths value va recurson s a standard step n the Baum-Welch algorthm and for ease of exposton, we assume that t has alread been computed (the detaled algorthm to compute t can be found n [11]). Now, startng wth an arbtrar settng of model parameters Θ, the EM algorthm terates the sequence of Equatons n Table 1 untl convergence: These steps gve a locall optmal parameter settng for our HMM. Tpcall, 15 teratons are enough for convergence. The number of states n the HMM has to be pcked b the user; we found that 10 states eld good results. 3.4 Algorthm Summar Algorthm 3 SUMMHMM INUT: Tweet corpus Z, tweet word vocabular V, desred number of tweets n, mnmum actvt threshold l OUTUT: Set of ke tweets T Learn Θ b teratng the equatons n Table 1 untl convergence Infer tme segments T S b the Vterb algorthm [11] T S = {s T S tweet volume n segment s > l% of all tweets} for each segment s T S do Z[s] = Z restrcted to tme s T s = SummAllT ext(z[s], V, n/ T S ) end for T = S T s 4. EXERIMENTS We frst descrbe the expermental setup, and then n successve sectons evaluate varous aspects of our proposed algorthms. 4.1 Expermental Setup In ths secton we descrbe the dataset used, the process of ground truth constructon, and the baselnes consdered n ths evaluaton. FINDING SORTS EVENTS IN TWITTER DATA. Our goal n ths paper was to summarze repeatng events. For our experments, we selected the sport of professonal Amercan Football. Football teams enjo enormous populart n the USA (see Fgures 1 and 2) and pla a large number of games wth each other over a ear, gvng us an deal test-bed to evaluate our proposed model. Snce our emphass s on summarzaton, all algorthms assume that the problem of searchng the tweets relevant to a user quer about a sports event has alread been solved. To smulate the perfect search process, we scanned Twtter feeds for the perod of Sep 12th, 2010 to Jan 24th, 2011 for tweets contanng the names of NFL teams: we notced that on Twtter users often appended ther posts about sport events wth hash-tags contanng the names of the teams nvolved. Hence, we collect the tweet-corpus relevant to the game between Green Ba ackers and Chcago Bears on Jan 23rd 2011 b fndng all tweets durng game-tme that contan ether #packers or #bears. We understand that our tweet-corpus s a subset of all the tweets pertnent to the sports event, and that a stronger search sstem mght fnd other relevant tweets; however, constructng such a sstem s beond the scope ths work. Even wth these constrants we obtaned over 440K tweets over 150 games for an average of around 1760 tweets per game. CLEANING THE TWEETS. The event-specfc tweets obtaned b the process descrbed above s ver nos. The two chef sources of nose are spam and tweets unrelated to the game. Spam-tweets can be easl removed snce the almost alwas have a URL n them. We also do not consder users that have less than 2 or greater than 100 tweets for an one football game. Fnall, we place smlar thresholds on number of occurrences of words: we remove (orter-stemmed) words that occur fewer than 5 tmes or n more than 90% of the tweets for the football game. Whle ths removes most of the nose, we are stll left

6 wth some tweets that do not strctl refer to the game under consderaton: tpcall, user rants about ther favorte teams, the game of football, or the world n general. After ths cleanng, we onl consder games wth greater than 1500 tweets from at least 100 ndependent users: we are left wth 53 of them. All experments henceforth are conducted on ths corpus. OUR AROACH AND BASELINES. Our approach and the baselnes were descrbed n detal n Secton 3, and here we wll gve the parameter settngs we used. SUMMALLTEXT: Ths approach constructs the game summar b fndng the set of tweets that are close to all others n the corpus (detals n Secton 3.1). We mplemented ths baselne b representng tweets va a TF-logIDF representaton and used Cosne smlart as the comparson measure. SUMMTIMEINT: Ths approach, descrbed n Secton 3.2, summarzes tweets as SUMMALLTEXT, but n each tme wndow separatel. It takes two parameters: we set t = 120secs and l = 1%. These parameters were set va evaluaton on a held-out valdaton set (10% of ground truth). SUMMHMM: Ths s our proposed HMM based approach that was descrbed n detal n Secton 3.3. We learn the underlng latent space of tweets b runnng 15 teratons wth K = 10 states. Later n ths secton we wll analze the structure of these learnt hdden states. After fndng the segments, the summar tweets are generated b callng SUMMTIMEINT wth parameters l = 1.5% (tuned through the use of a held-out 10% valdaton set). MANUAL GROUND TRUTH CONSTRUCTION. Our proposed approach and baselnes all output a set of tweets the consder as a good summar of the game. In order to evaluate these approaches we had human edtors manuall label ever tweet output b them. Each output tweet was matched wth the happenngs n the game and labeled as Comment-la, Comment- Game, or Comment-General. To be labeled as Comment-la the tweet had to explctl descrbe and occur wth a few mnutes of the pla n queston. The specfc tpe of pla (touchdown, feldgoal, ntercepton, fumble, etc.), was also noted n the label. These tweets were also assgned addtonal labels f the gave extra nformaton, lke current score, number of ards, and other contextual nformaton (Comment-la-Detals), or names of the plaers nvolved (Comment-la-Names). Tweets labeled Comment-Game tpcall descrbed the state of the game at that pont n tme, such as the current score or reports of plaer njures etc., and can be consdered hghl useful n a game summar. Tweets labeled Comment-General were statements that were not related to the football game under consderaton. A total of 2175 tweets were manuall labeled ths wa. 4.2 Constructng la-b-la Summar Here we evaluate our approach (SUMMHMM) and baselnes, SUMM- TIMEINT and SUMMALLTEXT, on the task of obtanng a useful pla-b-pla summar of football games. For these results we use the set of 2175 manuall labeled tweets descrbed n Secton 4.1. To report performance we resort to measures that are standard n research n nformaton retreval. We want an deal game summar to contan all mportant plas n football, lke touchdowns, feld goals, nterceptons, and fumbles. Hence, we defne RECALL of an approach as the fracton of such mportant plas n a game that are found b t. Further, we want the deal game summar to nclude as few rrelevant tweets as possble: the screen real-estate to dspla the summar and the user s attenton are both lmted. Hence, we defne the RECISION of an approach as the fracton of ts output tweets that are labeled Comment-la or Comment-Game. As s clear as an approach outputs more tweets tpcall the RECISION wll become smaller and the RECALL wll ncrease. Hence, we report RECISION Fgure 4: RECISION-RECALL curves of our approach and baselnes. To produce the curve the number of tweets was vared from n ncrements of 10. Around the operatng pont of 30 tweets, SUMMHMM has a RECISION@30 = 0.5 and a RECALL@30 = and RECALL b varng the number of output tweets from n ncrements of 10. EVALUATION AT OERATING OINT. In Fgure 4 we plot the RECISION-RECALL curves of our approach (SUMMHMM) and the baselne methods. Frst thng to notce s that performance of SUMMHMM domnates the performance of the SUMMTIMEINT and SUMMALLTEXT over the whole set of operatng ponts. It s dffcult to show the full set of 70 output tweets to users, owng to lmted user attenton and screen real-estate, the operatng pont of a deploed sstem s lkel to be lower. In the more realstc operatng ranges the performance of SUMMHMM s sgnfcantl hgher than both SUMMTIMEINT and SUMMALL- TEXT. In the mddle of that range n terms of RECISION@20 and RECALL@20, SUMMHMM s 25% and 16% better, respectvel, than the nearest compettor. RECALL OF DETAILS OF LAYS. Here we compare the performance of SUMMHMM and baselnes n terms of whether the output tweets gve context and detals around the pla. In Fgure 5 we plot the measure RECALL@30 on the task of retrevng Comment-la, Comment-la-Detals, and Comment- la-names labeled tweets. To be clear, RECALL@30 for Comment- la-detals s the fracton of mportant plas n a game that are matched b some top-30 tweet that has been labeled Comment-la- Detals b the edtors. If an mportant pla s matched wth some tweet n the top-30 that s onl labeled Comment-la or Comment- la-names, then t does not count towards RECALL@30 for Comment- la-detals. From the fgure, frst note that SUMMHMM outperforms the baselnes n all three tasks. In fact, as the tasks become harder the performance dfference between SUMMHMM and the nearest compettor s hgher as well: SUMMHMM s beats others b more than 33% on fndng Comment-la-Detals and b 26% on fndng Comment-la-Names. Second, note that the performance of SUMMHMM on fndng tweets labeled Comment-la-Detals and Comment-la-Names s lower than fndng tweets that are labeled Comment-la. Ths s due to the fact that whle twtter users are remarkabl consstent on how the refer to plas such as touchdowns and feld-goals, there s a much larger varablt when the refer to detals such number of ards ganed or names of plaers nvolved. laer names n partcular lend themselves to msspellngs and abbrevatons as well as ambgutes; most twtter users are postng through moble devces. As an extreme example, apart from hs correct name, the Ravens plaer T J Houshmandzadeh s referred to as tj, tjh,

7 Fgure 5: RECALL performance of our approach and baselnes on fndng tweets wth context and detals. As we can see the task of fndng tweets labeled Comment-la-Detals and Comment-la-Names s more dffcult and the performance mprovement due to SUMMHMM s hgher. Fgure 6: RECALL performance of our approach and baselnes on fndng tweets about dfferent tpes of plas. As we can see, scorng plas such as touchdowns and feld-goals are easer to fnd than others. houzhma, houshma, houshmoundzadeh n the tweets. Also, there are 19 plaers n 14 teams wth the name Jackson. Ths extremel varablt and ambgut n the names s the reason the RE- CALL performance on retrevng the name s low. As future work we wll consder specal algorthms for resolvng some of these ssues surroundng names before deplong a sstem lke ths. RECALL OF TYES OF LAYS. Here we dscuss the ablt of the varous approaches n fndng tweets that match dfferent tpes of plas n the game. In Fgure 6 we plot the RECALL@30 measure when restrcted to fndng just the plas of the tpes lsted on the x-axs. As we can see all methods are strong when retrevng kes touchdown-plas snce these are the prmar scorng mechansms n Amercan Football and precsel the plas whch generate most user tweets. However, a second crtcal scorng mechansm, feld-goal-plas, proves much harder for baselnes whle SUMMHMM detects 50% of them n the top-30 output tweets. The fracton of touchdown-plas detected n the top-30 tweets s around 60% whle the number rses to almost 90% n the top-70 tweets. An observaton about a ke result n Fgure 6 gves nsght nto a strength and a weakness of our approach. As we can see, SUMMHMM performs sgnfcantl worse on the task of fndng ntercepton-plas than the baselnes. Ths s because our approach makes a crtcal assumpton that after segmentaton va HMM there s but one ke pla wthn each segment. Ths s often a strength snce ths lets us use the segment specfc models to retreve the pla, but can sometmes turn nto a weakness when the assumpton s volated. In the case of ntercepton-plas, the are oftentmes followed mmedatel b other scorng plas: n our data around 45% of nterceptonplas were followed wthn mnutes b touchdown-plas or feldgoal-plas. Hence, the assumpton made b SUMMHMM s volated State label Top state-specfc words Top (state, game)-specfc words TOUCHDOWN stop, catch, drve, great, pass mark_sanchez FIELD-GOAL mss, kck, kcker, no_good, score nck_folk INTERCETION nt, throw, pck, touchdown, defense mark_sanchez DEFENSE & FUMBLE sack, good, punt, stop, block darrelle_revs ENALTY & FUMBLE challenge, run, hold, punt, call darrelle_revs Table 2: The label and top words from 5 hdden states learnt b SUMMHMM. The state label n column 1 s pcked from among the top state-specfc words lsted n column 2. Due to pauct of space we onl gve the top (state, game)-specfc word for each hdden state averaged over all games of the NFL team NY Jets. n the case of ntercepton-plas. One wa to tackle ths ssue s ensure dverst amongst tweets when selectng them from a segment; we plan to attempt ths n future work. REASONS FOR RECISION@30 =0.5. As we ponted out earler n the deploment operatng regon of returnng around 30 tweets, the recson = 0.5. Whle ths seems a lttle low as a number, upon nspecton of the tweets output b the varous approaches t s clear that the edtors enforced a ver strct standard n the judgments. A tweet was deemed relevant onl f t referred to a specfc dentfable pla n the game. For example, tweets encouragng the teams such as let s convert on 3rd down #packers and gvng personal opnons on the game such as hate that was the rght call b the refs #packers were deemed rrelevant. We beleve that ncludng these tweets nto the summar adds color and drama to t and hence these tweets should be scored relevant. However, the true worth of an of these presentatons can onl be determned once we observe how users nteract wth them; we hope to run ths amended evaluaton as part of the future work. 4.3 Anecdotal Evdence and Dscusson We have seen that SUMMHMM s ver effectve n generatng pla-b-pla summares of Amercan Football games. In ths secton we provde anecdotal evdence of the sstem s performance as well as dscussons of ssues n summarzng events usng Twtter updates. HIDDEN STATES OF THE LEARNT HMM. In Table 2 we dspla 5 of the 10 hdden states used n learnng the HMM and gve ther top state-specfc as well as state-game-specfc words. The rest of the states ether acted as duplcates of these states, or had unnterpretable term dstrbutons; ths s common n learnng hdden state models where the number of underlng hdden states has to be guessed. Frst, notce that these states do correspond to the dfferent tpes of plas n Amercan Football. Second, whle the state-specfc words capture the words that tweets use to descrbe the pla n general, the words specfc to the (state, game) pars are tpcall plaer names that change from game to game. Column three of Table 2 shows the top (state, game) words averaged over all games of the NY Jets team. The results make complete sense as Mark Sanchez s the team s quarterback (or man offensve plaer) and Darrelle Revs seems to be ther man defensve plaer. Nck Folk, the team s kcker s the top word for the state FIELD-GOAL. Fnall, some of related pla-tpes are represented b the same underlng hdden state. Ths happens because ether these plas are referred to usng smlar words, or the same plaers partcpate n them, or the often occur ver close together n tme. Hence, we see that SUMMHMM fnds a cohesve underlng hdden structure of Football games. GENERALIZATION TO OTHER TYES OF EVENTS. SUMMHMM has been desgned and mplemented to generate good summares for repeatng sports events, but ts functonalt s a strct super-set of those needed to work on other tpes of events. Amercan Football has a ver dscrete nature wth sgnfcant events well separated n clock-tme, whle other sports lke soccer have a much more contnuous game-flow; ths can be challengng for the baselnes descrbed

8 n ths paper. We have tested SUMMHMM on soccer matches from World Cup 2010 and obtaned results smlar to Amercan Football games. Other tpes of events such as musc festvals and awardshows, or even sudden news events such as the Tunsan revoluton, can beneft from a Twtter-based summar though such events are not repeated often. We expect that SUMMHMM wll be able to handle these events, however, some of ts features wll be under-utlzed. As future work we would lke to appl our approach to these other tpes of events. RELIANCE ON EFFECTIVE SEARCH. In ths work we assume that the ntal process of event detecton and retreval of a relevant set of tweets s accomplshed: SUMMHMM works on ths set of relevant tweets. There exst man works on event detecton on Twtter [2, 12, 3] and search engnes currentl do a decent job of retrevng relevant tweets; hence, we beleve ths s a reasonable assumpton. However, even f the retreved set of tweets s nos, we beleve that SUMMHMM wll be able to construct clean summares. Ths s because SUMMHMM constructs summares b formng a underlng model of the event. Ths model s most heavl nfluenced b the major trends n the tweet data; an outler tweet (erroneousl retreved) that does not correspond to one of the larger tweet clusters s gnored. We see ths n the data used for experments n Secton 4.1. The data collecton process s ver smple and just looks for tweets contanng #team-name ; ths results n the collecton of man rrelevant tweets that are not removed even b our cleanng approach. However, these tweets end up beng gnored b SUMMHMM when learnng the model and constructng the summares. 5. RELATED WORK Here we descrbe some of the related exstng research work and dscuss how our work dffers from t. MICROBLOGGING AND TWITTER. There has been much recent nterest on dentfng and then trackng the evoluton of events on Twtter and other socal meda webstes, e.g., dscussons about an earthquake on twtter [12], detectng new events (also called frst stores) n the tweet-stream [10], vsualzng the evoluton of tags [5], and other events on Flckr, Youtube, and Facebook [2, 3]. The problem has also been approached from the pont of vew of effcenc: [8] propose ndexng and compresson technques to speed up event detecton wthout sacrfcng detecton accurac. However, to the best of our knowledge, we are the frst to stud and summarze events usng user update on Twtter. We assume that the event detecton has alread been performed, possbl usng one of the aforementoned technques; our goal s to collate all the nformaton n the tweets and present a summarzed tmelne of the event. SUMMARIZATION. Summarzaton of text documents has been well studed n the IR communt. A common method s based on computng relevance scores for each sentence n the document and then pckng from among the best [6]. More complex methods are based on latent semantc analss, hdden markov models, deep natural language analss, among others [6, 4]. However, these are prmarl amed at standalone webpages and documents. Whle the summarzaton of a sequence of tweets can be seen as a form of text summarzaton, there are nuances that are not consdered b most pror algorthms. For nstance, each tweet s ver short (at most 140 characters), spellng and grammatcal mstakes abound, sentences are often just phrases, and some tweets nclude snppets of others ( re-tweets ). In addton, the sequence of tweets for an event s created b the actvt of a multtude of users nstead of a sngle author, and hence there s lttle contnut n the tweet-stream. Facng such condtons, researchers have turned to smpler heurstcs, such as b usng the words before and after known topc-specfc phrases to expand the set of phrases relevant for summarzaton [13]. We also emprcall found the smplest methods to work best, and n our experments we used such a summarzaton method. The man contrbuton of our work s n modelng the underlng hdden structure of events; most an off-the-shelf summarzaton method can then be then used to extract the mportant tweets to construct the summar. SEQUENTIAL MODELING. There are a varet of methods to model sequence nformaton. Change-pont models tr to detect nstants of tme when there s some marked change n the behavor of the sequence, e.g., the ntenst wth whch observatons arrve suddenl changes [9, 7]. Thus, change-ponts gve a segmentaton of the sequence nto chunks of dfferent ntenstes. Another wa to segment the sequence s on the bass of dfferences n dstrbuton of the observatons themselves. A classcal technque for ths s the Hdden Markov Model (HMM) [11, 14], whch posts the exstence of an underlng process that transtons through a sequence of latent states, wth observatons beng generated ndependentl b each state n the sequence. HMMs have been extremel successful n areas as vared as speech recognton, gesture recognton, and bonformatcs. As we showed n Secton 3, the tweet-streams for sports events are ver burst, and segmentaton nto burst epsodes s necessar before accurate summarzaton can be performed. Segmentaton based on the words and phrases appearng n the tweet-stream s just as crtcal, snce a sngle hgh-ntenst burst can actuall be composed of man sub-events, each wth ts own vocabular and word dstrbuton. In addton, for repeatng events (such as sports), modelng the tweets from all the events can lead to better accurac than modelng each event n solaton. None of the aforementoned technques acheves all of these goals. Hence, we developed a modfed HMM that can account for (a) shfts n ntenst, and (b) shfts n the language model, both over tme n a gven event, and also across events. 6. SUMMARY AND FUTURE WORK In ths work we tackled the problem of constructng real-tme summares of events from twtter tweets. We proposed an approach based on learnng an underlng hdden state representaton of an event. Through experments on large scale data on Amercan Football games we showed that SUMMHMM clearl outperforms strong baselnes on the pla-b-pla summar constructon task, and learns a underlng structure of the sport that makes ntutve sense. As future drectons of research we would lke to test SUMMHMM on long-runnng one-shot events such as festvals and award shows, and to provde summares for mportant but unpredctable events such as revolutons and natural dsasters. Fnall, we have not et evaluated the summares generated b our approach n real-tme on search engne users; ths s somethng we hope to do n the future. 7. REFERENCES [1] R. A. Baeza-Yates and B. Rbero-Neto. Modern Informaton Retreval. Addson-Wesle Longman, Boston, MA, [2] H. Becker, M. Naaman, and L. Gravano. Learnng smlart metrcs for event dentfcaton n socal meda. In WSDM, [3] L. Chen and A. Ro. Event detecton from flckr data through wavelet-based spatal analss. In CIKM, [4] D. Das and A. Martns. A surve on automatc text summarzaton, [5] M. Dubnko, R. Kumar, J. Magnan, J. Novak,. Raghavan, and A. Tomkns. Vsualzng tags over tme. In WWW, [6] Y. Gong and X. Lu. Generc text summarzaton usng relevance measure and latent semantc analss. In 24th SIGIR, pages 19 25, [7] J. Klenberg. Burst and herarchcal structure n streams. In KDD, [8] G. Luo, C. Tang, and. Yu. Resource-adaptve real-tme new event detecton. In SIGMOD, [9] H. Mannla and M. Salmenkv. Fndng smple ntenst descrptons from event sequence data. In KDD, 2001.

9 [10] S. etrovć, M. Osborne, and V. Lavrenko. Streamng frst stor detecton wth applcaton to twtter. In HLT, [11] L. Rabner. A tutoral on hmm and selected applcatons n speech recognton. roc. of the IEEE, 77(2): , [12] T. Sakak, M. Okazak, and Y. Matsuo. Earthquake shakes twtter users: Real-tme event detecton b socal sensors. In WWW, [13] B. Sharf, M.-A. Hutton, and J. Kalta. Summarzng mcroblogs automatcall. In HLT, [14]. Wang, H. Wang, M. Lu, and W. Wang. An algorthmc approach to event summarzaton. In SIGMOD, 2010.