Fully Heterogeneous Collective Regression

Transcription

1 Fully Heterogeneous Collective Regression ABSTRACT Davi J. Lietka Department of Computer Science Unite States Naval Acaemy Annapolis, Marylan Prior work has emonstrate that multiple methos for link-base classification (LBC) can substantially improve accuracy when the noes of interest are interconnecte. To ate, however, very little work has consiere how methos for LBC coul be applie in omains that require continuous, rather than categorical, preictions. In aition, prior work with LBC has learne only one preictive moel to use for all noes of a given type, but some omains exhibit significant noe iversity that is not well-suite to this approach. In response, we introuce fully heterogeneous collective regression (FHCR), a new metho that learns noe-specific moels from ata an uses these moels to jointly preict continuous outputs. We apply FHCR to a voting preiction task, an create novel correlation-base links that outperform alternative methos. In aition, we introuce multiple new methos for inferring continuous outputs that can incorporate link-base information, an show that regression-specific methos base on Bayesian inference outperform the naive approach of inserting regression into existing LBC methos. Overall, we emonstrate the viability of the new FHCR paraigm by proucing results that are comparable or better than those of previous link-unaware methos, yet are at least two orers of magnitue faster. 1 INTRODUCTION Increasingly, many important omains in the worl can be viewe as networks of linke noes, such as people in online social networks or webpages connecte by hyperlinks. To leverage these links for preiction an analysis tasks, Machine Learning researchers have evelope multiple techniques for link-base classification (LBC) [4, 5, 10, 12, 14, 16]. While LBC can substantially improve preiction accuracy in some omains, current limitations greatly restrict its applicability when use to evaluate heterogeneous omains (e.g., when the collection of noes uner stuy are actually rawn from multiple populations). Aitionally, traitional LBC preicts only categorical outputs, while link-base regression to preict continuous outputs has been left largely unexplore. One such application that requires continuous outputs involves elections. Preicting the voting outcome of national or regional elections is a challenging yet important problem, an has great implications for regional an international security. As just one example, how well can the national outcome of an election be preicte, given past voting history an some incomplete ay of voting results? A recent stuy by Etter et al. [3], using Swiss referenum outcomes, reporte high accuracy, even when only 5% of voting regions ha reporte results. This stuy use a matrix factorization approach to implicitly leverage the correlation Luke K. McDowell Department of Computer Science Unite States Naval Acaemy Annapolis, Marylan lmcowel@usna.eu present between nearby regions. They i not, however, consier formulating the regions as a network. This paper presents the first extension of LBC methos to noespecific preictive moels an continuous outputs, proucing fully heterogeneous collective regression (FHCR). To emonstrate the effectiveness of this approach, we apply it to the voting preiction task of Etter et al. We show that our link-base approach can be highly effective, even though the ata initially contains no links. Our contributions are as follows: We introuce a new paraigm, Fully Heterogeneous Collective Regression (FHCR), for preictions base on relational ata. This new paraigm enables for the first time the application of link-base inference algorithms to omains with highly heterogeneous objects an continuous outputs. We introuce multiple new methos for inferring continuous outputs, for both the initial bootstrap problem where preictions must be mae without links, an for the collective inference step where links are use. In both cases, we emonstrate that novel applications of Bayesian inference often lea to improve voting preiction accuracy, especially when relatively little information is known about a particular vote. In aition, we show how this inference can elegantly combine link-base information with information about object features, yieling further accuracy gains. We create novel methos for link creation base on historical correlations, an emonstrate that for voting preiction they generally yiel more accurate results than simpler methos base on geographic proximity. We perform extensive evaluations of FHCR on the voting preiction task. By combining our new methos, we achieve voting preiction results that are competitive with or even improve upon those use by the prior stuy, but execute times faster than previous methos. Below, Section 2 introuces relevant backgroun on LBC an Section 3 iscusses work specifically relate to FHCR. Section 4 explains the voting task of Etter et al. an explains why FHCR is appropriate for this task. Section 5 gives an overview of FHCR an explains the key steps neee to apply it to voting preiction. Section 6 presents our experimental results, an Section 7 conclues. 2 BACKGROUND ON LBC Assume we are given a graph G = (V, E, X,Y,C) where V is a set of noes, E is a set of eges (links), each x i X is an attribute vector for a noe v i V, each Y i Y is a label variable for v i, an C is the set of possible labels. We are also given a set of known values Y K for noes V K V, so that Y K = {y i v i V K }. Then

2 a common link-base classification (LBC) task is to infer Y U, the values of Y i for the remaining noes V U with unknown values (V U = V \ V K ). For example, given a (partially-labele) set of interlinke university webpages, consier the task of preicting whether each page belongs to a professor or a stuent. The simplest (non-lbc) approach woul be to learn a moel that preicts the label for page v base solely on the attributes of v, such as the presence or absence of certain wors. An LBC approach woul instea combine these self attributes with some relational features, which are base on the labels of pages that link to v. For instance, it might construct a relational feature such as Count the number of v s neighbors with label Stuent. However, this is challenging, because some labels are unknown an must be estimate, typically with an iterative process of collective inference [5], such as Gibbs sampling, belief propagation, or ICA (Iterative Classification Algorithm) [16]. For this paper, the most relevant collective inference approach is ICA, a simple, popular, an effective algorithm [10, 12, 16]. ICA first preicts a label for every noe in V U using only self attributes. It then constructs aitional relational features X R using the known an preicte noe labels (Y K an Y U ), an re-preicts labels for V U using both self attributes an X R. This process of feature computation an preiction is repeate, e.g., until convergence. 3 RELATED WORK While there has been substantial prior work on collective classification [4, 5, 10, 12, 14, 16], only a few exceptions have consiere collective regression [1, 9, 18]. Loglisci et al. [9] an Aloah & Neville [1] both use aggregation functions like mean or meian to aggregate information from linke neighbors into relational features that are then provie to a regression tree moel for actual preiction. Aloah & Neville preict total movie revenue using a ataset from the IMDB atabase, while Loglisci et al. preict various target values for eight small (less than 1000 noes) social an spatial atasets. For collective inference, Loglisci et al. essentially use ICA where the classification moel is replace with a regression tree. Aloah & Neville use this approach as well, but also o M rouns of graient boosting, where subsequent rouns learn to preict the amount of resiual error that remains from previous rouns. In contrast, Zhang et al. s NetCycle [18] approach primarily concerns classification, again using ICA with relational features create by aggregation. However, their paper inclues one set of experiments that perform collective regression instea of collective classification. They o so by replacing the classification moel with support vector regression, without other moifications specific to regression. Our work with FHCR iffers from the above works with collective regression in two primary ways. First, we introuce an use a fully heterogeneous approach, where every noe has its own regression moel. This allows us to leverage the historical ata of each region, accounting for regional ifferences. In contrast, the above methos use a single moel for all noes, with the exception of NetCycle, whose partially heterogeneous approach learns 2-4 ifferent moels for the entire ataset (e.g., one moel for authors, another for conferences, etc.). Secon, all of the above approaches use a form of ICA with regression, where relational 2 features somehow aggregate the information about linke neighbors, an then these features (together with attributes about each noe) are provie as input to a link-unaware regression moel (such as regression trees). We also consier this metho (which we call RegressICA), but our results show that instea constructing a new approach that explicitly leverages the continuous nature of the target values, an their linke neighbors, yiels better results. In particular, our results later show that methos base on Bayesian inference, using Gaussians to represent the learne conitional istributions, lea to more accurate results while maintaining computational tractability. For work focuse on classification, rather than regression, there have been a number of other stuies that use partially heterogeneous approaches, similar to that use by NetCycle. For instance, Neville an Jensen [15] stuies link-base classification for a movies omain that inclues movies, proucers, an stuios. They learn ifferent moels for each type of noe (e.g., for movie, proucer, an stuio), as oppose to the noe-specific moels that we use for our fully heterogeneous approach. In contrast, some other work (e.g., [7, 17]) in omains where there are multiple noe types (such as papers, authors, an conferences) still learns only a single moel (e.g., for papers), but the relational features are constructe base on meta paths that can traverse multiple types of noes. Thus, this work also escribes itself as heterogeneous. Our particular task of voting preiction coul potentially be accomplishe through methos base on recommener systems an/or matrix factorization. For instance, Koren et al. [8] explores the use of collaborative filtering to generate user-specific movie recommenations. Koren et al. shows how latent factorization methos can be use to infer unintuitive relationships between ifferent sets of factors to make more accurate preictions; Etter et al. uses some such methos an we compare against them. Note, however, that the omains motivating such methos typically have very sparsely-labele ata, as a user is unlikely to, for instance, have watche a majority of all films in the target omain. In contrast, we have a ense set of historical voting recors, which naturally leas to a ifferent set of learning challenges. 4 TARGET PROBLEM & DISCUSSION Because relational ata is so commonplace, there are numerous applications for which the FHCR paraigm coul be useful for making more accurate preictions. One of these applications is the preiction of voting outcomes. For this task, continuous, noespecific, an online results are esirable. First, continuous results are esire because, while a referenum ultimately has a single binary result ( passe or faile ), various stakeholers woul like to be able to preict with much greater fielity whether a referenum will pass with strong support, weak support, or not pass at all. Secon, noe-specific preictive moels are esire because the noes uner stuy are regions that exhibit significant iversity, such that using a single moel for all regions woul substantially ecrease accuracy. For FHCR, this approach is feasible wherever we have extensive information about each noe (region) or small groups of noes, as is true with our historical voting recors. Finally, we woul like online results, meaning that as the results for more regions are reporte (e.g., on the ay of an election), we

3 shoul be able to quickly make new, more precise preictions for the remaining unreporte regions. In this paper we apply FHCR to preict voting outcomes on the ataset use by Etter et al. [3]. The ataset contains voting outcomes for 281 Swiss national referenums over a perio of 34 years. As with Etter et al., we use first 231 votes for training an the final 50 votes for testing. Results are reporte (as the proportion of yes votes, between 0 an 1) for each of the 2352 regions (municipalities) in Switzerlan. Each region is escribe by 25 per-region emographic features x (population, population ensity, language spoken, location, etc.), while each vote n has 13 per-vote features w n that are the recommenations ( for, against, or no recommenation) mae by major political parties. Since online preiction is esire, the task is to preict the yes proportion, on some test vote, for every region, given the voting results for some number of known regions (which are assume to have alreay reporte their results, or to have high-quality estimates from surveying). The vote an region features can be use for preiction. However, to compensate for the varying popularity of ifferent votes, Etter et al. mean-centers each training an test vote, using the true national average for the training votes an the estimate national average (compute using only the known regions) for the test votes. Thus, the possible values for preiction range from -1.0 to 1.0. We use this same approach. Etter et al. use a four factor moel to preict continuous outputs for regional votes. This moel incorporate a bias term for each vote, a regression term base on region features, a regression term base on vote features, an a matrix factorization moel base off of latent features an the set of votes. However, Etter et al. i not use any LBC or collective regression. In particular, Etter et al. oes not treat the ata as a graph or explicitly use links. 5 PREDICTION WITH FHCR FHCR follows a pattern similar to that of ICA as escribe in Section 2: initial preictions are mae using features, then a collective inference proceure iteratively upates those preictions for each region base on its relevant features an the preictions of linke regions. This section thus stuies the following key questions require to make FHCR effective: (1) How to bootstrap the initial preictions to provie a baseline for our inference? (2) How shoul links be compute, so as to connect the regions into a useful graph? (3) Given the initial preictions an informative links, how can we perform effective collective inference? 5.1 Methos for Bootstrap Preiction Before collective inference can be use for a particular test vote n, we must compute a set of initial bootstrap preictions y (0) for each region. These are compute using the vote an/or region features, but without the use of links (see summary in Table 1). Some methos use by Etter et al. are suitable for this task, so we consier those first: LIN(v) - linear regression base only on the vote features. We train each region s moel by comparing the 13 political party recommenations w m for each training vote m to that region s voting 3 Table 1: Summary of features an regression types use in various bootstrap moels. Moel Use vote feats? Use region feats? Combination Metho LIN(v) Yes No N/A LIN(r) No Yes N/A Joint(r,v) Yes Yes Alternating Least Squares BayesD+Inpt(r,v) Yes Yes Approximate Bayesian Inference BayesG+Inpt(r,v) Yes Yes Exact Bayesian Inference JBG-Ensemble Yes Yes ALS an Bayesian Inference outcome, yieling the following preictions for test vote n: y (0) = γt w n where γ is a weight vector specific to region. LIN(r) - regression base only on the region features. For test vote n, we use a vote-specific moel that takes x (the 25 features of region ) as input, with coefficients β n learne using only the known region results for vote n. Then for inference, y (0) = βt n x. Joint(r,v) - regression base on the region features an the vote features. This approach uses alternating least squares to learn a joint moel: y (0) = βt n x + γ T w n. Etter et al. calle this moel LIN(r)+LIN(v). New methos: Of the above methos, Etter et al. generally showe that using both sets of features performe best. However, the linear combination approach of Joint(r,v) is not necessarily optimal. In response, we create three new bootstrap methos that combine region an vote features in ifferent ways: BayesD+Inpt(r,v) - This metho performs Bayesian Inference using conitional istributions for the vote an region features, learne inepenently. First, we use Bayes rule to calculate the probability that the preiction for a particular region, y, is a certain value y, given the region features x an the vote features w n : P(y = y x,w n ) = P(x,w n y = y) P(y = y) P(x,w n ) P(x y = y) P(w n y = y) P(y = y) P(y = y x ) P(y = y w n ) P(y = y) where the secon line assumes conitional inepenence an the thir line re-applies Bayes rule to the first two terms. To estimate P(y = y w n ), we apply the LIN(v) moel to the training ata. Then, we generate a iscrete histogram (we use 500 uniform-size bins) base on the error between the true training values an preictions mae with LIN(v), for all regions an training votes. This learns a single histogram for all regions, which can then be applie uring inference for a particular region (as P(y = y w n )) by shifting the mean of the histogram to the value preicte by LIN(v). P(y = y x ) uses a similar same process but with LIN(r), where learning uses all training votes but only the regions that will be known uring inference. Finally, the actual bootstrap step fins the most likely value of y via Equation 1 an a iscrete estimate of the following integral: (1) 1 y (0) = y P(y = y x,w n )y. (2) 1 BayesG+Inpt(r,v) - this metho also uses Bayesian Inference to leverage the vote an region features, via Equation 1. However,

4 by approximating the conitional probabilities as Gaussians, we enable exact inference, instea of resorting to a iscrete approximation (hence, the G vice D in the name). In particular, we substitute normal istributions for each probability, yieling the following: P(y = y x,w n ) N(βT n x, σ 2 r ) N(γ T w n, σ 2 v ) N(0, σ 2 prior ). In this equation, each mean is compute base on the results from LIN(r) or LIN(v), or is known to be zero (ue to mean-centering of the ata). The variances (σr 2, σv 2, an σprior 2 ) can be estimate by comparing preictions to actual values in the training ata. With this form, the most likely value for y can be compute exactly, because the prouct of two Gaussians f an д is also a Gaussian [2, 6], with the following mean an variance: µ f σд 2 + µ д σ f 2 µ f д = σ f 2 + σ д 2 an σ 2 f д = σ 2 f σ 2 д σ 2 f + σ 2 д JBG-Ensemble - a combination of the previous two methos. We average the preictions generate by the Joint(r,v) an the BayesG+Inpt(r,v) moels, potentially mitigating the impact of outliers preicte by either moel. Later results will show this can be very effective, while remaining computationally tractable. 5.2 Effective Link Generation Target uses of FHCR may involve noes, such as people in a social network, for which links alreay exist or are obvious. In other cases, as with our voting omain, there may be no explicit links, in which case a key task is to construct informative links to connect the noes (i.e., regions), in a way that facilitates subsequent inference. In our ata, regions are ientifie by latitue/longitue, an thus one natural iea is to link all regions that are within a certain istance, such as 5 kilometers. Distance can also be inverte an scale to prouce link weights, so that closer regions have more influence. Section 6 shows that both of these ieas can be effective. For voting preiction, however, we hypothesize that linking regions base on similar voting histories woul prove even more effective than links base on geographic proximity. Thus, we examine the 231 training votes an compute the Pearson correlation coefficient between all pairs of regions. We then consier methos base on linking, for each region, the k most-similar regions, or all regions with a correlation greater than some threshol (e.g., 0.80). 5.3 Collective (Link-base) Regression Regression techniques are well establishe, but collective regression requires some kin of iterative proceure to allow the preicte an known values to propagate throughout the links. In each case, the initial preicte values (y (0) ) are set via a bootstrap metho from Section 5.1. For inference, we first consier two baseline methos, which are simple extensions of existing LBC methos to regression: RegressICA - This metho uses ICA, escribe in Section 2, but with an unerlying moel of regression rather than classification. Preiction for region is base on s vote features an, as a relational feature, the average preicte value for s neighbors. For each iteration i (after bootstrap), preictions are compute as (3) 4 j L y (i 1) j L y (i) = f (w n, ) where f is a learne (linear) regression moel an L is the set of regions that link to region. We use 10 iterations; more i not improve accuracy. When links are weighte, the neighbor average is a weighte computation, but for ease of explication we omit all such link weight etails in this section. WeighteVec - Instea of using the neighbor average as a feature for regression, this metho uses the neighbor average as its actual preiction. Thus, y (i) = j L y (i 1) j. This metho is analogous to the weighte vote relational neighbor (WVRN) metho L of Macskassy an Provost [11], but aapte for regression instea of classification. New methos: We also create the following new methos for collective inference: BayesD+Links - this metho seeks to use Bayesian Inference to estimate the most likely preiction for y, given only the preicte (an known) values of region s linke neighbors, y L. This can be compute as follows: P(y = y y L ) = P(y = y) P(y L y = y) P(y L ) P(y = y) P(y j y = y) j L where the last step assumes conitional inepenence of the neighbors values given y. In general, estimating P(y j y = y) from the training ata coul be a challenging task. However, we conjecture that for our purposes this probability can be approximate as a function solely of the ifference between y j an y. 1 That iea yiels P(y = y y L ) P(y = y) ϕ(y j y ) (4) j L where ϕ(x) is some arbitrary function. To approximate ϕ(x), we first calculate, for each region an training vote, the ifference between the mean-centere voting outcome for an those of each of its linke neighbors. We then a these values to a histogram (again using 500 uniform-size bins) to prouce a iscrete estimate of ϕ(x) (share across all regions). For inference, we can then compute the most likely value of P(y = y y L ) via an approximation of 1 y (i) = y P(y = y y (i 1) L )y. 1 This is similar to our previous use of Equation 2, but now preicting with s neighbors instea of s features. BayesG+Links - This metho is similar to BayesD+Links, but approximates the link-base conitional probabilities with Gaussians, enabling exact inference via variants of Equation 3 (see [2]). With this approach, Equation 4 becomes P(y = y y L ) N(0, σ 2 prior ) j L N(y j, σ 2 links ) The first term is the prior, with assume mean zero, an variance estimate from the training ata. The link-base Gaussian for region 1 For example, we estimate that the likelihoo of y j being 0.3 given y is 0.25 is approximately the same as the likelihoo of y j being 0.4 given y is This woul be approximately true if similar (linke) regions ten to vote in the same ways.

5 j, which replaces ϕ(y j y ), uses a mean of y j (the preicte or known value for j), an variance σ 2 links (base on fitting ϕ(y j y ) to a Gaussian with the training votes). This metho has the avantage of yieling much faster inference than BayesD+Links (e.g., about 40 times faster in our experiments). In aition, it avois errors that result from iscretization. However, o the Gaussians actually represent the conitional probabilities of Equation 4 in a way that is effective for inference? Section 6 evaluates their impact. Incorporating features: The methos iscusse above (except for RegressICA) have use only the link information for their inference. The following new methos all seek to improve preiction accuracy by also exploiting the vote an/or region features. WeighteVec+Delta - this metho uses the WeighteVec approach, but moifie to also leverage vote-base features. In particular, we learn a region-specific elta correction, base on the training error between each region s true value an the value preicte by WeighteVec, yieling j y (i) = L y (i 1) j + f L (w n ) where f uses the vote features for preiction. This relates to the resiual learning of Aloah & Neville [1]. BayesD+Inpt(r,v)+Links - while similar to BayesD+Links, this metho takes region an vote features into consieration in aition to the links. Following a similar erivation as before yiels P(y = y x,w n,y L ) P(y =y x ) P(y =y w n ) j L ϕ(y j y ) P(y =y) where the three conitional probabilities can be estimate as previously iscusse. BayesG+Inpt(r,v)+Links - this metho uses the same approach as above, but approximates the probability istributions as Gaussians. Therefore, P(y = y x,w n,y L ) N(βT n x, σ 2 r ) N(γ T wn, σ 2 v ) j L N(y j, σ 2 links ) N(0, σ 2 pr ior ) (6) BayesG+Joint(r,v)+Links - this metho uses an inference strategy similar to BayesG+Inpt(r,v)+Links. Above, we assume conitional inepenence for the conitional probabilities associate with the region an the vote features. However, the region an vote features may have interactions. Thus, instea of using two separate Gaussians for these features, we use one Gaussian base on the Joint(r,v) implementation of the region an vote features. Collapsing these two terms into one also causes the enominator to cancel, yieling P(y = y x,w n,y L ) N(β T n x + γ T w n, σ 2 joint ) j L N(y j, σ 2 links ) BayesG+JGB-Ensemble+Links - this metho is nearly ientical to the previous metho, except that the mean use for the first Gaussian is compute base on the preiction from the JBG- Ensemble bootstrap, with variance base on averaging the variance of the two Gaussians arising from the two parts in the ensemble. (5) 5 Table 2: Average RMSE after running only the bootstrap step, with no collective inference. The best result for each column is in bol. Number of Known Regions, N k Bootstrap Metho LIN(v) LIN(r) Joint(r,v) BayesD+Inpt(r,v) BayesG+Inpt(r,v) JBG-Ensemble RESULTS Below, we stuy how to use FHCR most effectively via appropriate choices for bootstrap, link generation, an collective inference. Our experiments replicate the experimental conitions use by Etter et al., using the ata an conitions escribe in Section 4. For a given test vote, the task is to preict the yes percentage for each region, given the results for some subset of regions (thus simulating a preiction task with partial ay of voting results); there are N K such known regions. For each trial, these regions are selecte by first choosing a ranom reveal orer for all 2352 regions. The first N k regions are consiere known, while the last 10% are reserve as evaluation regions. Performance is evaluate by measuring the root mean square error (RMSE) over the evaluation regions, average over 60 trials. Each trial uses the same reveal orer for all methos, an we use Etter et al. s original coe to prouce results with their methos. 6.1 Bootstrap Table 2 shows the RMSE values vs. the number of known regions (N k ) as we vary the bootstrap metho; lower values are better. In general, error ecreases as N k increase for two reasons. First, methos that use the region features, such as LIN(r), use the known values for the given test vote to learn their classifier, an learning with more such results ecreases error. However, error also ecreases for LIN(v), whose classifier is learne solely from votebase features with 231 fully-observe training votes (an thus not irectly affecte by N k ), because the preicte mean for test vote n is estimate from the N k known regions (see Section 4). A larger N k prouces a better estimate, which helps all moels. We first consier the three methos from Etter et al. (first three rows of Table 2). When there are very few known regions (N K < 10), using only the vote features, with LIN(v), performs best. In contrast, using only the region features has higher error, ue to the small number of regions available for training LIN(r), as iscusse above. LIN(r) improves rapily as N k increases, so that it outperforms LIN(v) for N k 10. However, using both the region an vote features, with Joint(r,v), performs even better, for N k 10. We next consier our three new bootstrap methos (last three rows of Table 2), which all use some Bayesian inference to combine the region an vote features. For all cases except N k = 1, the Gaussian approximation with exact inference (BayesG+Inpt(r,v)) yiels better RMSE than the iscrete version (BayesD+Inpt(r,v)). Moreover, combining these preictions with those of Joint(r,v) (with JBG-Ensemble) yiels even lower error, when N K 50. Overall, the last two rows of Table 2 show that our new methos combining the vote an region features with Bayesian inference succee in reucing the error, compare to the best previous alternatives,

6 LIN(v) an Joint(r,v). This emonstrates the general utility of the Bayesian inference metho for combining isparate sources of information, as will be further emonstrate in Section Link Generation Table 3 shows the results of collective inference, with ifferent link generation methos. All cases use 10 iterations of RegressICA; the next section consiers ifferent collective inference strategies. Base on the results of the prior section, we use JBG-Ensemble for bootstrap, though results (not shown) using BayesG+Inpt(r,v) instea yiele very similar results. The top section of Table 3 shows results with proximity-base links, with istance threshols of 5, 10, an 20 kilometers. Of these, Prox-10km works best (except for N k = 1), by striking a goo balance between having too many links (with a large threshol) an having too few. Aing link weighting base on nearness (with Prox-10km-Wt) oes not consistently improve the performance. The secon section of Table 3 shows results with links chosen by correlation, e.g., linking all pairs of regions with a correlation of at least 0.60 or Picking many links with the lower threshol (Corr- 0.60) performs better when many regions are known (N k > 500); in this case, the high value of N k reuces preiction uncertainty, an averaging over many links is helpful. In contrast, the higher threshol (Corr-0.80) performs better when N k 500, inicating that having a smaller number of strongly correlate links is more effective. The problem with this approach, for some cases, is that using a high threshol causes some regions to have only a few links. In response, the next two rows establish a minimum number of links for each region. This metho links region to all other regions with correlation at least 0.80, or, if there are not enough such regions to meet the minimum, to the top 10 or 20 most strongly correlate regions. With this strategy, Corr-0.80-Min-10 almost always improves over Corr-0.80 an Corr For instance, for N k = 1000 the RMSE improves from 6.40 with Corr-0.80 to to Encourage by the success of setting a minimum number of links, the thir section of Table 3 thus consiers linking each region to its top k = 10 or k = 100 correlate other regions. Using 10 links for each region (Corr-10) almost always performs a little better than the methos iscusse above, an substantially better than using 100 links (Corr-100). For instance, for N k = 1000, Corr-10 has RMSE of 5.46, while Corr-100 has RMSE of The last two rows of Table 3 show results where the links are weighte base on the inter-region correlation values. While leaing to marginal improvements in some cases, we fin that weighting links in this case makes minimal ifference overall. For Corr-10-Wt, this is because there is little variation in the correlation values when only 10 links are chosen, so link weighting makes little ifference. For Corr-100-Wt, the ifferences (vs. Corr-100) are larger but still quite small. Future work shoul consier whether a ifferent correlation formula (instea of Pearson s) might provie values more useful for this task, an make the final results less sensitive to the number of links chosen. Overall, we fin that we fin that Corr-10-Wt is the most effective general strategy. While Prox-10km (an sometimes Prox-5km) performs best for N k 10, Corr-10-Wt is still an effective strategy for these values of N k. Thus, we chose Corr-10-Wt as our linking strategy for the next sections. 6 Table 3: Average RMSE of various link generation strategies when running RegressICA with the Ensemble bootstrap. Number of Known Regions, N k Linking Metho Prox-20km Prox-10km Prox-5km Prox-10km-Wt Corr Corr Corr-0.80-Min Corr-0.80-Min Corr Corr Corr-10-Wt Corr-100-Wt Collective Inference Table 4(a) shows results of varying the collective inference metho, using JBG-Ensemble for bootstrap an Corr-10-Wt for link generation. All methos use 10 iteration of collective inference. RegressICA an WeighteVec can be consiere baseline methos for FHCR that are aapte from typical approaches to linkbase classification. Since RegressICA uses features an links, it performs better than WeighteVec (which use only links), except for N K 5. Nonetheless, WeighteVec also obtains accuracy close to that of RegressICA when N k is very high; this is reminiscent of WVRN s strong classification performance, compare to more complex methos, for more ensely-labele graphs [11]. The secon section of Table 4(a) compares the three link only methos: WeighteVec, BayesD+Links, an BayesG+Links. 2 In every case, our use of Bayesian inference ecreases the error compare to the simple averaging approach of WeighteVec (by 0.03 to 0.19), an the Gaussian approximation with BayesG+Links further reuces error compare to the iscrete approximation with BayesD+Links (by 0.01 to 0.10). In aition to reucing error, inference with the Gaussians also executes about 40 times faster than with the iscrete version. The thir section of Table 4(a) as the use of vote an/or region features to the three link-only methos. With Weighte- Vec+Delta, this consistently reuces the error, sometimes substantially (e.g., from 5.94 to 5.50 when N k = 500). With the four methos base on BayesD an BayesG, however, the results are more mute. While aing feature information uring inference (with Inpt(r,v), Joint(r,v), or JBG-Ensemble) generally improves accuracy compare to BayesD+Links an BayesG+Links, the error improves by at most 0.23 an sometimes gets slightly worse. Thus, while aing feature information generally improves the collective inference, with the Bayesian inference it is surprising that this aition is not more helpful. The next section aresses this issue. 6.4 Improving Inference via Variance Scaling The previous section observe that incorporating the region an vote features into our inference was beneficial. However, the features were significantly less helpful for Bayesian inference than they were for WeighteVec. We conjecture that this inicates that the Bayesian methos are not balancing, as effectively, the 2 Note that these link only inference methos still use the feature information uring bootstrap.

7 Table 4: RMSE for various collective inference methos, using JBG- Ensemble bootstrap an Corr-10-Wt linking. Within each column, we bol the best results. Part (a) of the table uses methos from Section 5.3, while part (b) as variance scaling (see Section 6.4). Number of Known Regions, N k Inference Metho (a) Propose FHCR methos (without variance scaling) RegressICA WeighteVec BayesD+Links BayesG+Links WeighteVec+Delta BayesD+Inpt(r,v)+Links BayesG+Inpt(r,v)+Links BayesG+Joint(r,v)+Links BayesG+JGB-Ensemble+Links (b) BayesG methos with aition of variance scaling BayesG+Inpt(r,v)+Links BayesG+Joint(r,v)+Links BayesG+JGB-Ensemble+Links link-base an the feature-base information. In aition, careful comparison with Table 2 shows that, so far, the best results with collective inference are actually worse than the best results with just bootstrap, when N K 10 (for instance, for N k = 5, BayesG+JGB-Ensemble+Links has RMSE of 8.46, but bootstraponly BayesG+Inpt(r,v) obtains 8.19). Thus, when there is more preiction uncertainty (ue to fewer known regions), the Bayesian collective inference is sometimes oing more harm than goo. We conjecture that the problem with the Bayesian methos is that they treat all linke neighbors as being equally reliable for preiction. In orer to improve our Bayesian inference, we must somehow teach it to iscriminate between certain an uncertain values, known an unknown neighbor preictions. This woul also allow a region s features to have more weight when its linke neighbors are less certain, which coul help for the small N K scenario. Fortunately, certain properties of our BayesG approach can enable us to introuce this iscrimination, as we now escribe. Equation 3 shows that, when combining two Gaussians, the Gaussian with the smaller variance will have more impact on the resulting mean. However, Equation 6 uses the same term, N(y j, σ 2 links ) for each neighbor. In this term, the mean for each neighbor varies (base on the preicte or known value y j ), but the variance is the same, regarless of the certainty of y j. This variance is learne from the training ata, where there are no preiction errors, an thus is appropriate for a link to a known region. When linking to a preicte region, however, this estimate likely unerestimates the true variance. Therefore, we propose to scale the variance use for unknown regions as follows: σ 2 linkst ou nknownreдions = K U σ 2 links where K U is a constant learne via cross-valiation. In particular, we evaluate the RMSE obtaine via collective inference on the last 15 votes of the training ata, using K U {2, 4, 8} an select the K U that minimizes RMSE on this valiation ata. We then use that K U for inference on the actual test ata an report results. With K U > 1, the larger variance will lessen the preictive influence of regions with uncertain preictions. Table 4(b) shows the results for three Bayesian inference variants with the aition of this variance scaling. As esire, the use of variance scaling leas to substantial error reuctions in many cases. 7 For instance, BayesG+Inpt(r,v)+Links improves from 8.48 to 8.16, for N k = 5, which is the best for any metho in Table 2 or Table 4. Overall, this eliminates the situation where bootstrap only results outperforme the best collective inference, an in general some form of Bayesian inference with variance scaling now performs best among the FHCR methos that we consier in Table 4, with two slight exceptions at N k = 100 an N k = Discussion an Comparison Table 5 compares our best results (using Bayesian inference with variance scaling) with the four methos that Etter et al. use for their in-epth comparisons. The first two methos are baselines: BIAS, which simply uses the average of all known results as its preiction, an the previously iscusse LIN(v), which uses regression on the vote features. The next two methos are those that Etter foune performe best: MF+GP(r) an MF+GP(r)+LIN(v). Both use matrix factorization (MF) combine with a Gaussian process (GP) for the region features. The left sie of Table 5 reports per-region RMSE results as use elsewhere in this paper, while the right sie shows the national binary error rate. To compute this metric, which Etter et al. also consiere, we first use the per-region preictions weighte by region population to compute an overall national preiction; values greater than 0.5 yiel a preiction of pass as the overall vote outcome. The error rate is then the fraction of votes that are incorrectly classifie as pass vs. fail. The error rate slightly increases when N K is very high; Etter et al. attributes this effect to weighting the final preiction by each region s population, as oppose to the actual ay of voting turnout, which is not a priori observable. Overall, Table 5 shows that FHCR is highly effective at the voting preiction task. In particular, our new methos (an Etter s) consistently ecrease the errors compare to the BIAS an LIN(v) baselines. Moreover, when N K is smaller, one of our new FHCR methos prouces the best results, both for per-region RMSE an the national error rate results. For per-region RMSE, our methos are somewhat better than Etter s (by ), when N k < 50, while they prouce competitive results (lagging by at most 0.41 RMSE) for larger values of N k. Interestingly, our methos o even better (relatively) on the national error rate results, suggesting that they ten to o especially well on more populous regions (which have a larger effect on the national outcome). In particular, while the Etter methos perform best when N K 500, for all N K < 500, our new BayesG+Inpt(r,v)+Links (with variance scaling) yiels the lowest national error of any metho. For instance, when only 100 regions have known results, this metho has an error rate of just 0.87%, while the best Etter metho has an error rate of 1.23%. Even more significantly, our new methos base on Bayesian inference are much faster than Etter s best methos. In particular, running on an Intel i7-4600m 2.90 GHz CPU, Etter et al. s MF+GP(r)+LIN(v) takes about 480 minutes to fully train an provie one trial of inference for one vote, while MF+GP(r) requires 3360 minutes. 3 In contrast, using our moels base on BayesG, incluing cross-valiation for variance scaling, requires only 4.25 minutes, a speeup of times vs. the Etter moels. Note 3 For both our coe an Etter et al. s, the runtime is ominate by training the moel, while inference runs much more quickly. The simpler MF+GP(r) took longer (vs. MF+GP(r)+LIN(v)) because more learning iterations were require.

8 Table 5: Comparison of the best results with FHCR (first three rows) vs. two baseline methos from Etter et al. (next two rows) an the best results from Etter et al. (last two rows). On left, we report average RMSE results for each region, as in the rest of the paper. On right, we show binary error percentages, where goal is to preict the overall outcome (pass/fail) for each vote. Within each column, we bol the best result. Per-region results (RMSE) National results (binary error percentage) Number of Known Regions, N k : New FHCR methos base on BayesG (with variance scaling) BayesG+Inpt(r,v)+Links BayesG+Joint(r,v)+Links BayesG+JGB-Ensemble+Links Methos from Etter et al. [3] BIAS LIN(v) MF+GP(r) MF+GP(r)+LIN(v) that this faster time is much more amenable to online an ynamic analysis, as useful for ay of voting preictions. Thus, compare to prior work, our new FHCR methos obtain comparable or better accuracy, especially when relatively few regions have reporte results, while executing at least two orers of magnitue faster. 7 CONCLUSION Link-base classification (LBC) has been shown to substantially improve accuracy in a variety of omains, but prior to this unertaking ha rarely been applie to continuous omains, an never to heterogeneous omains where rich temporal/historical information coul potentially enable noe-specific moels. Our results show that extening the LBC paraigm to fully heterogeneous collective regression (FHCR) inee aresses these limitations of LBC. We have evelope techniques that are efficient an effective. In particular, we introuce novel methos that combine featurebase an link-base information using Bayesian inference. We showe that these methos always outperforme extensions of LBC methos like ICA, an almost always outperforme multiple versions base on neighbor averaging (e.g., as with WVRN [11]). Moreover, we emonstrate results that are comparable to or slightly better than the previous link-unaware methos use by Etter et al., especially when the results from only a small number of regions are known yet our methos are at least two orers of magnitue faster. This greatly facilitates online preiction of results, for example for upate ay of voting results, or for pre-voting result forecasting, where resources are available to extensively canvas only a small fraction of voting regions. Future work shoul consier further improvements to these methos. For instance, we emonstrate that variance scaling was highly effective at reucing error for our methos base on Bayesian inference Future work shoul stuy further refinements, such as varying the effective variance use for each region base on an estimate confience for its current preiction [13]. Some aspects of our propose methos are specific to the voting preiction task that we use, such as combining information from two istinct feature sets (the per vote an per region features) that require istinct learning strategies. Other omains where FHCR coul be useful may not have such feature iversity. However, our methos using Bayesian inference to combine the information from a varying number of linke neighbors, an to combine that information with feature-base preictions, are applicable in other settings, such as per-region sales preiction, weather forecasting, 8 crow-sourcing preictions [18], an other time-varying social phenomena. Future work shoul apply FHCR to these omains. ACKNOWLEDGEMENTS We thank Ewar Raff an the anonymous referees for comments that helpe improve this work. Grants/resources from ONR an the DoD HPC Moernization Office supporte this work in part. REFERENCES [1] Iman Aloah an Jennifer Neville Combining Graient Boosting Machines with Collective Inference to Preict Continuous Values. In Proceeings of the 6th International Workshop on Statistical Relational AI. [2] Paul Bromiley Proucts an convolutions of Gaussian probability ensity functions. Tina-Vision Memo 3, 4 (2003), 1. [3] Vincent Etter, Mohamma Emtiyaz Khan, Matthias Grossglauser, an Patrick Thiran Online Collaborative Preiction of Regional Vote Results. In Proc. of DSAA. IEEE, [4] Brian Gallagher, Hanghang Tong, Tina Eliassi-Ra, an Christos Faloutsos Using ghost eges for classification in sparsely labele networks. In Proc. of KDD [5] Davi Jensen, Jennifer Neville, an Brian Gallagher Why collective inference improves relational classification. In Proc. of KDD. ACM, [6] Ruolph Emil Kalman A new approach to linear filtering an preiction problems. Journal of Basic Engineering 82, 1 (1960), [7] Xiangnan Kong, Philip S Yu, Ying Ding, an Davi J Wil Meta path-base collective classification in heterogeneous information networks. In Proc. of CIKM [8] Yehua Koren, Robert Bell, an Chris Volinsky Matrix factorization techniques for recommener systems. Computer 42, 8 (2009). [9] Corrao Loglisci, Annalisa Appice, an Donato Malerba Collective regression for hanling autocorrelation of network ata in a transuctive setting. Journal of Intelligent Information Systems 46, 3 (2016), [10] Qing Lu an Lise Getoor Link-base classification. In Proc. of ICML [11] Sofus A Macskassy an Foster Provost Classification in networke ata: A toolkit an a univariate case stuy. Journal of Machine Learning Research 8, May (2007), [12] Luke McDowell an Davi Aha Semi-supervise collective classification via hybri label regularization. In Proc. of ICML. [13] Luke K. McDowell, Kalyan M. Gupta, an Davi W. Aha Cautious collective classification. Journal of Machine Learning Research 10 (2009), [14] Jennifer Neville an Davi Jensen Iterative Classification in Relational Data. In Proc. of the Workshop on Learning Statistical Moels from Relational Data at AAAI [15] Jennifer Neville an Davi Jensen Relational epenency networks. Journal of Machine Learning Research 8, Mar (2007), [16] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, an Tina Eliassi-Ra Collective classification in network ata. AI magazine 29, 3 (2008), 93. [17] Yizhou Zhang, Yun Xiong, Xiangnan Kong, Shanshan Li, Jinhong Mi, an Yangyong Zhu Deep Collective Classification in Heterogeneous Information Networks. In Proc. of WWW [18] Yizhou Zhang, Yun Xiong, Xiangnan Kong, an Yangyong Zhu Netcycle: Collective evolution inference in heterogeneous information networks. In Proc. of KDD