A Meta-Analysis of Role Ambiguity and Role Conflict on IS Professional Job Satisfaction

Transcription

1 A Meta-Analysis of Role Ambiguity and Role Conflict on IS Professional Job Satisfaction Yide Shen CIS Department Robinson College of Business Georgia State University Atlanta, GA, USA Abstract There have been numerous IS studies on the topic of role ambiguity (RA) and role conflict (RC). The need for resolving the disagreement in understanding the impact of RA and RC on job satisfaction within an IS context motivated the current research. Employing Hunter and Schmidt s (1990) meta-analysis method, this study synthesizes empirical IS study results from twelve independent samples. The results revealed that both RA and RC are negatively correlated with IS professional s job satisfaction across diverse studies. Compared with non-is related jobs, IS professionals are not necessarily more- or less-susceptible to the negative impacts of RA and RC on job satisfaction. Furthermore, the unexplained variance in the study effect sizes indicates the existence of potential moderating variables. Finally, an unexpected but noteworthy contribution of the current research is to raise the issue of misuse of the term meta-analysis within the IS research community. 1. Introduction With the proliferation of computers and information systems in organizations, information technology (IT) professionals are playing increasingly critical roles in supporting organizational operations and attaining strategic goals. Hence, understanding the factors that influence IS professionals job satisfaction is an area which concerns both IS managers and IS researchers. The importance of this issue is reinforced by research showing that job satisfaction is the most important antecedent of IS employees intention to stay in an organization and thus, the most important determinant of IS personnel turnover [3]. Additionally, decreased job satisfaction often leads to increased absenteeism, ill health, and grievances [40], which impairs the quality of work life of IS professionals. Among the variables expected to influence job satisfaction, two role stressors role ambiguity (RA) and role conflict (RC) have captured the attention of numerous researchers. The results from both the organizational behavior and IS literatures suggest that RA and RC are generally negatively related to job satisfaction, however the strength and significance level of these relationships are not always consistent. Especially in IS research, we lack a definitive conclusion about the relative importance of RA and RC. The need for resolving the disagreement in understanding the impact of RA and RC on job satisfaction within an IS context motivated the current research. Given the broad range of empirical IS research on this topic, a formal meta-analysis is useful to accumulate results across many independent studies and synthesize the statistical data from multiple sources. In addition, due to the uniqueness and rapid changes occurring in the IT environment, it is worthwhile to empirically examine the applicability of role stressor-job satisfaction relationships in IS settings. This is the objective of this paper: to accumulate empirical results regarding the relationship between RA/RC and IS professionals job satisfaction across independent studies, to examine the relative importance of the two role stressors, to compare them with those from a broad range of job occupations in the organizational behavior literature, and investigate the variables that may moderate these relationships. The remainder of the paper is structured as follows. Section 2 reviews the concepts of RA and RC as they have been described in the management literature and develops hypotheses for this study. Sections 3 and 4 describe the meta-analysis method employed in this study. Section 5 presents the result of our analysis. The paper concludes by discussing the results from the quantitative analysis and their implications for IS personnel researchers and IS scholars, more broadly. 2. Literature Review and Hypotheses A role is most typically defined as a set of expectations about behavior for a position in a social structure [47, p. 155]. RA refers to the degree to which a given job is lacking in terms of 1) the predictability of 1

2 responses to one s behavior, and 2) the existence or clarity of behavior requirements [47]. RC is defined as the degree of incongruity or incompatibility in the expectations or requirements communicated to a focal person [4, p. 93]. It is generally believed that RA and RC result in lower job satisfaction. Consistent with the theoretical basis, most of the empirical results reported medium to high negative correlations between either RA or RC and job satisfaction, while only two studies have shown positive correlations [25, 32]. In line with the goals of this research to accumulate empirical results regarding the relationship between RA/RC and IS professionals job satisfaction across independent studies we propose the first two hypotheses: Hypothesis 1: RA and job satisfaction will be negatively correlated among IS professionals. Hypothesis 2: RC and job satisfaction will be negatively correlated among IS professionals. Another inconsistency that has been observed is the relative impact of either RA or RC on job satisfaction. For instance, by studying a sample of 229 IS professionals from nine companies in various industries (e.g., insurance, banking, manufacturing, etc.), Baroudi [2] found that RA is a more influential variable, accounting for 22% of the variance in job satisfaction, compared to RC, which accounted for only 7% of the variance in job satisfaction. In contrast, based on their study of 109 IS managers from a wide variety of industries (e.g., banking, insurance, manufacturing, etc.), Li and Shani [37] concluded that: Role ambiguity, having the lowest mean composite score [2.9] among the four stressors perceived by the IS managers [work overload (5.0), role conflict (4.5), jobinduced anxiety (3.2), and role ambiguity (2.9)], is significantly influenced by many organizational contextual as well as job satisfaction factors. Ivancevich, Napier, and Wetherbe (1983) had a similar finding that role ambiguity was not a major stressor perceived by the IS personnel; it was the fifth highest mean score among the seven stressor scales used in their study [37, p. 123]. However, a closer review of these two studies reveals some differences in terms of the research design. First, Baroudi s sample consisted of IS professionals from a range of job types (such as applications programmers, programmer/analysts, analysts, and project leaders), while the subjects in Li and Shani s sample were all IS managers. Second, the two studies used the same measure for RA and RC [47], but they used different measures for job satisfaction (Job Descriptive Index [50] vs. a homegrown scale [16]). Thus, it is impossible to conclude whether the different impacts of RA and RC on job satisfaction are due to the nature of these two constructs, or whether the contradictory results are introduced by these other artifacts. Based on above discussion, hypothesis 3 compares the magnitude of the negative effects of RA and RC on job satisfaction: Hypothesis 3: The correlation between RA and job satisfaction will differ from the correlation between RC and job satisfaction among IS professionals. Two meta-analysis studies from the organizational behavior literature have identified a generally negative relationship between both RA and RC and job satisfaction in many different types of occupations, although the strength of the relationships varied across different occupational samples [10, 28]. Kahn and colleagues [30] study of various jobs revealed that the level of RC and RA were high among people whose jobs involved innovation or boundary spanning (i.e., coping with people outside the person s work group). This is exactly the nature of most types of job in the IS work environment. For example, rapid changes in technological innovation require IS employees to update their knowledge and skills frequently. System analysts need to work with cross- departmental and even crossorganizational stakeholders. These characteristics indicate that the problem of high RC and RA especially pertains to IS professionals: The evidence is highly suggestive, therefore, that systems professional with their high need for achievement, autonomy, and cognition, and low need for change are particularly prone to the adverse effects of role conflict and role ambiguity [2, p. 342]. On the other hand, it is possible that because role stressors are inherent within IS work environments, workers that have been selected for and socialized to work within the IS field have a relatively high tolerance for RA and RC [23]. King and Sethi [31] described how the process of socialization may make IS professionals accustomed to accepting such challenges in their jobs. Thus, even though the levels of RA and RC may be high for IS professionals, their high levels of tolerance for role ambiguity and role conflict may help to moderate the negative effects that would otherwise occur, in terms of low job satisfaction. Based on above discussion, hypothesis 4 compares the negative effects of RA and RC between IS professionals and other non-is job types: Hypothesis 4a: The strength of the relationship between RA and job satisfaction will be similar among IS professionals and other job types. Hypothesis 4b: The strength of the relationship between RC and job satisfaction will be similar among IS professionals and other job types. 2

3 3. Meta-Analysis Method The term meta-analysis was coined by Glass [11, p. 3] to refer to the analysis of analyses the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. The main thrust of meta-analysis methods is to accumulate empirical results across independent studies and provide a more accurate representation of population characteristics. Over the years, meta-analysis has become a legitimate statistical tool to integrate empirical research findings in many disciplines, such as medicine, education, and psychology [19]. Hunter and Schmidt s [18] psychometric meta-analysis is the primary method employed in this study. The focus of Hunter and Schmidt s method is on effect sizes, which represent correlational measures of effect. The advantage of Hunter and Schmidt s approach is that it provides techniques to correct for artifactual and methodological problems that may distort effect sizes from the true score values. In survey research, correlations (Pearson s r) are the most frequently-used measure of effect size, and the focus of the current research. (In contrast, controlled laboratory studies employ other types of effect sizes to characterize the differences between experimental and control group effects, such as Cohen s d.) Following the processes suggested by Hunter and Schmidt [18], our meta-analysis consists of three major phases: literature search, coding of study information (both methods and results), and data analysis. Each phase is discussed in detail in the following sections Literature Search A literature research was performed primarily on electronic sources (ABI/Inform, EBSCO Business Source Premier, ACM Digital Library, and ScienceDirect), as well as several conference proceedings (ICIS, HICSS, AMCIS, and ACM SIG CPR) using the keywords role ambiguity, role conflict, job satisfaction, stress, role stress, burnout, and work exhaustion. Since the focus of this research is on IS professionals, the term information systems and information technology were used in combination with the above keywords to restrict the search results. We also reviewed the reference list of the articles retrieved to identify other related papers. The search yielded 40 papers related to RA, RC, and job satisfaction in an IS context. Methodological details and correlational results were then coded based on these papers Coding of Study Information A total of 25 papers were excluded from preliminary screening for various reasons. Since the focus of this study is on IS professionals, four papers examining end users or other non-is occupational types were excluded. Meta-analysis requires data regarding correlations and sample sizes in order to compute a weighted correlation between independent and dependent variables, thus, another eighteen papers were excluded because they either were theoretical papers without any empirical data (two papers) or didn t report correlations for either of the role stressor variables and job satisfaction (sixteen papers). In addition, three other papers that did not study job satisfaction were also excluded. Following Viswesvaran and Ones s [51] advice, methodological weaknesses were not treated as grounds to eliminate any studies. For example, if a paper reported a weak result for internal reliability (Cronbach alpha) for RA or RC, rather than excluding the study, we incorporated the information about the (weak) reliability metrics, and used Hunter and Schmidt s procedures to correct for this methodological weakness (and others). The remaining fifteen papers were then coded for the following information: sample size, Pearson s correlations between the role stressors and job satisfaction, and reliability statistics (Cronbach s alpha). During the coding process, we found that five data samples were each used in two different papers, respectively. To avoid double counting of the same sample, only one paper for each sample was included in our analysis. Furthermore, two papers separated their sample into two sub-groups. One of these studies [25] split the sample randomly, using one sub-sample to validate their instruments and another sub-sample to test their research model. The two subgroups from this study were thus treated as independent samples in our analysis. The other study [26] divided the sample according to the mean value of a specific personality variable (Type A versus Type B personality behavior), and found a moderating effect of this variable. Hunter and Schmidt [18] suggested using subgroup correlations for the two subgroups separately if there is evidence of a moderating effect. The two subgroups from the latter study were also treated as independent samples. This left us with a final sample size of ten papers consisting of twelve independent samples (Table 1). Of the twelve samples, all of them reported correlations between RA and job satisfaction, but only ten of the twelve samples reported correlations between RC and job satisfaction. The meta-analysis for each set of variables is discussed in the following section. 3

4 Table 1. Final sample for meta-analysis r Cronbach's Alpha Study Sample # Sample N Role Role Job RA JS RC JS Ambiguity Conflict Satisfaction 1 Goldstein Rockart Bostrom ~ ~ ~ Ivancevich, Napier, and ~ 3 Wetherbe ~ ~ 4 Li Shani Igbaria, Parasuraman, and Badawy Guimaraes Igbaria Igbaria Chidambaram Baroudi Lee King and Xia * RA JS: correlation between RA and Job Satisfaction; RC JS: correlation between RC and Job Satisfaction. 4. Data Analysis There are four steps to conducting a meta-analysis: correcting for any study artifacts, calculating effect sizes, computing the Fail-Safe N, and evaluating the homogeneity or heterogeneity of sample effects. The first step is to correct for measurement error. With the reliability statistics for both independent and dependent variables, measurement error was corrected using the formula provided by Hunter and Schmidt [18]. In cases where the reliability values were missing in the samples statistics, the averaged Cronbach s alpha value was used instead. After correcting for measurement error, the sampling error was also corrected. The second step is to calculate the effect size by averaging the reported correlations, weighted by sample size [18, p. 150]. The formula for effect size is r = [ N iri ] N i, where r i is the correlation after correcting for measurement error in study i and N i is the number of subjects in study i. The third step is to calculate the Fail-Safe N, which is a way to address the file-drawer problem the sampling bias in meta-analysis which assumes that an unknown number of studies with effect sizes of zero were not included in the analysis because they have either not been submitted for publication or have been rejected [48]. That is to say, the actual mean effect size may not be as high as the one generated by meta-analysis, due to the fact any studies reporting weak or zero effect sizes are less likely to be published, and thus less likely to be included in the meta-analysis. However, the combined study results can be highly significant statistically even though the mean effect size is small or even tiny [18, p. 512]. Hunter and Schmidt [18] further provided the formula to calculate Fail-Safe N, which indicates the number of missing studies with zero effect size that would have to exist to bring the mean effect size down to some specific level (0.1 in this study). The lower the value of the Fail-Safe N (e.g., a value = 2), then the more uncertainty that exists with regard to whether the mean effect size may be biased by the number of studies are buried in the file drawer, due to weak, non-existent, or contrary effects. The fourth step examines whether the results are consistent (homogeneous) across studies, or whether there are large variations in the pattern of correlations. The meta-analysis results are meaningful only if the various samples are homogeneous. The purpose of the homogeneity test is then to inspect whether any variations in results across studies are due to sampling error, or due to possible moderator variables [18]. Two criteria were applied to evaluate homogeneity of the sample unexplained variance and the Q test of homogeneity. As a rule of thumb, Hunter and Schmidt [18] suggested that if the fraction of unexplained variance is less than 25%, this variance can be neglected (i.e., due to some unmeasurable errors). Otherwise, if the percentage of unexplained variance is greater than 25%, there might be potential moderators that affect the correlation. The second criterion is the Q test of homogeneity. If the Q test value is not significant, then there is no true variation across studies; but if it is significant, it is reasonable to conclude that there are moderating variables influencing the correlations found from various studies. 4

5 Table 2. Results of meta-analysis Correlate k Total N Mean Effect Size % Unexplained 95% C.I. Fail-Safe N Variance in sample Homogeneity Test Lower Higher correlations Q p RA* --> JS % RC* --> JS % * RA JS: correlation between RA and Job Satisfaction; RC JS: correlation between RC and Job Satisfaction. 5. Results and Discussion 5.1. Hypotheses Testing Hypothesis 1 and 2 Supported. The results of the meta-analysis are summarized in Table 2. Table 2 shows that the mean effect size for the correlation of RA and job satisfaction is , and that between RC and job satisfaction is In general, both role stressor variables RA and RC are negatively correlated with job satisfaction. Thus, hypotheses 1 and 2 are both supported. Based on Cohen s [7] standard effect size levels of 0.10, 0.30, and 0.50 as small, medium, and large, respectively, the strength of the relationship between RA and job satisfaction (-0.271) is close to medium, while that between RC and job satisfaction (-0.243) is somewhat weaker than medium. Another principle to judge the significance of the mean effect size is based on confidence intervals. According to [39], if the confidence interval does not include the value of zero, then the mean effect size is significant at the level specified by the confidence intervals. The confidence interval shown in Table 2 indicate that both of the mean effect sizes are significant at the p<0.05 level, because zero is not included in the confidence intervals. Furthermore, according to Hunter and Schmidt [18], even when the computed mean effect size is relatively small, the cumulated study results can still be statistically significant. When we examine the Fail-Safe N for each relationship, we would need to locate 21 additional studies with zero-effect sizes in order to reduce the calculated mean correlation of between RA and job satisfaction to 0.1 (the threshold for weak correlation) when our result is calculated based on the 12 studies that we did find and code. Moreover, 15 studies with zeroeffect sizes would be required to reduce the mean effect size from correlation across studies to 0.1 when we calculated the mean correlation between RC and job satisfaction based on 10 studies. In summary, both RA and RC are negatively correlated with job satisfaction with medium strength (for RA) and somewhat less than medium strength (for RC). The correlations are significant at 0.05 level. The large Fail- Safe N values (21 and 15 for RA and RC, respectively) are a sign of considerably strong results Hypothesis 3 Not Supported. We hypothesized that the magnitude of the negative effects of RA and RC on job satisfaction would be different. To test this hypothesis, the non-parametric Wilcoxon paired signed rank test was conducted based on the corrected correlation coefficient multiplied by the sample sizes for each individual study. The result indicates that the correlations of RA and RC with job satisfaction (after correcting for sampling error and measurement error), did not differ significantly from each other (Z=-1.078, p=0.139). Thus, RA and RC have a statistically indistinguishable negative effect on job satisfaction. Hypothesis 3 is disconfirmed Hypothesis 4 Supported. The two meta-analysis studies [10, 28] from the organizational behavior literature provide a comparative basis for Hypothesis 4. Jackson and Schuler s [28] meta-analysis study included 96 papers none of which featured IS professionals. Fisher and Gitelson s [10] study reviewed 43 studies and 59 independent samples. Even though they didn t provide population information for the samples, a review of the studies titles included in their bibliography did not indicate the presence of any IS-related samples. Therefore, the synthesized effect sizes from these two studies represent the relationship between RA/RC and job satisfaction across various types of occupations (which appear not to have included IS employees). The mean effect sizes in the current study are, by and large, comparable to these two prior meta-analysis studies. More specifically, the correlation between RA and job satisfaction from this study (-0.271) falls between the effect sizes obtained from the two prior meta-analyses published in the organizational behavior literature (-0.25 and -0.30, respectively). In contrast, the correlation between RC and job satisfaction in our study (-0.243) is weaker than the effect sizes from both of these prior studies (-0.35 and -0.31, respectively). To further test the statistical significance of above comparisons, a non-parametric Mann-Whitney U test was 5

6 performed. Since only one of the organizational behavior studies [28] provided the original correlations for each individual study in its meta-analysis, the correlations from the current study are compared only with those from Jackson and Schuler [28]. 1 The results of the Mann-Whitney U test indicated that the correlations between RA and job satisfaction among IS professionals are not statistically different from other job types (Z = , p=0.961). Similarly, correlations between RC and job satisfaction did not differ statistically between IS professionals and other job types (Z = , p=0.371). Thus, both Hypothesis 4a and 4b are supported. While the mean effect size for the correlation between RC and job satisfaction was slightly weaker (-0.243), compared to the two prior meta-analyses for other job types (-0.35 and -0.31)., this difference was not statistically significant.. These results indicate that even though most IS jobs require both innovation and boundary spanning [2], which, in turn, introduce highs level of RA and RC, respectively [30], the negative impacts of RA and RC are not necessarily stronger among IS professionals than for other occupations. One potential explanation of this phenomenon is that IS professionals are selected for (or have learned to have) high tolerance for ambiguity and role conflict [23, 31] Moderating Effect The unexplained variances in both of the correlations are higher than the recommended threshold of 25%. This indicates that sampling error only accounts for a small percentage of the variance in sample correlations (2.92% for RA, 3.51% for RC). Thus, there may be one or more moderators having influence on the correlations found from various studies. The highly significant homogeneity Q test results also corroborates this conclusion. It is also consistent with that of the two organizational behavior meta-analyses [10, 28], namely, theoretically meaningful moderators should be included to explain the variance of both correlations across the samples. On the other hand, the unexplained variance and the significant Q test result from this study are both even higher than the results obtained from general job settings. However, considering that we conducted this analysis only among IS professional and excluded the samples of IT end users, we have, to some extent, controlled for variations in job type. The significant results from the homogeneity test suggest that large variations among the broad category of IS professionals may exist. This may indicate that a further specification of the job types of IS 1 In addition, because the reliability statistics (Cronbach s alpha) for each individual study are not shown in Jackson and Schuler s (1985) paper, our analysis is based on the original correlations, instead of the weighted correlations after correcting for sampling error and measurement error. professionals are required, rather than simply aggregating all employees together under the general category of IS professionals. According to Hunter and Schmidt [18], subgroup analysis can be performed to detect possible moderator variables. The hypothesized moderator variable can be used to split the sample into two or more subsamples. The same meta-analysis procedures described above then can be carried out for each subset independently. For example, the data could be analyzed separately for all samples that consisted of systems analysts, and another analysis for samples consisting of project managers. If the variance of the effect sizes in each subset is lower than that for the data as a whole, it shows that the variable used to split the samples is a moderator variable which causes the differences in average effect size. Some IS researchers have inspected potential moderators (job involvement, self-esteem, gender) for the relationship between role stressors and job satisfaction. Job involvement is defined as the psychological identification of an individual with a specific job [25, pp. 177]. Based on a sample of 464 IS professionals holding various job positions, Igbaria and colleagues [24] found that the impact of role stressors on IS employees differed significantly among different levels of job involvement (low, moderate, and high). Particularly, the negative consequences of both RA and RC are stronger among the moderate and high job involvement subgroups than for the low-involvement subgroup. The authors concluded that: these findings are analogous to those documented in the work-family literature that high levels of job involvement and family involvement heighten the level of workfamily conflict experienced by individuals and its aversive effects on individuals well-being [24, p. 193]. Prior research has also found self-esteem to moderate the relationship between role stressors and psychological and physical strain [29]. One IS study based on a sample of 124 system developers revealed that self-esteem moderates the relationship between role stress and job satisfaction [44]. Nevertheless, the authors didn t mention the direction of the moderating effect in either their hypotheses or results. Igbaria and Chidambaram [23] investigated the impact of gender on various aspects of IS professionals among 348 members of the Data Processing Management Association (DPMA) professional association. Their results showed that women and men reported no significant differences in role stressors. After controlling for gender, they found that both RA and RC were still significant predictors of job satisfaction. Another potential moderator that may differentiate among the observed results is the diversity in job satisfaction measurements used for different studies. Among the ten studies included in our meta-analysis, job satisfaction was measured by five different scales: 6

7 Hackman and Oldham's (1976) Job Diagnostic Survey (JDS), Smith, Kendall and Hulin's (1969) Job Descriptive Index (JDI), the Minnesota Satisfaction Questionnaire Short Form (1967), measures from Hoy [16], and revised measures from Hoppock (1935). Some of these instruments feature various dimensions of job satisfaction (e.g., satisfaction with pay, promotional opportunities, coworkers, and the job itself) Yes only four studies reported the correlation between role stressors and the specific subdimensions of their job satisfaction scales [50]. Such a lack of specific results may inhibit the comparison between general job satisfaction and these facets of job satisfaction, which have been shown to differ in the earlier meta-analysis studies [10, 28]. For example, Jackson and Schuler s study showed that it is necessary to explore moderators for the relationship between RC and general job satisfaction and also for RC and satisfaction with advancement opportunities, while there was no evidence of moderators for other sub-types of satisfaction (satisfaction with supervision, work itself, co-workers, and pay), because the percentage of unexplained variance was close to zero. Despite our identification of several potential moderator variables, due to the minimal sample size required in each sub-sample for conducting a moderator analysis, and to the lack of necessary information for each potential moderator, we were unable to conduct further moderator analyses with the available data. 6. Conclusion 6.1. Summary of Research Findings This paper presents the use of meta-analysis as a statistical technique to integrate empirical IS research results regarding the relationship between RA/RC and IS professionals job satisfaction across independent studies. In general, the results of this study found that both of the role stressor variables RA and RC are negatively correlated with IS professional s job satisfaction across diverse studies. The mean effect size for RA is more strongly correlated with job satisfaction (-0.271) than the analogous correlation between RC and job satisfaction ( ), although the difference between them was not statistically significant. Compared with non-is related jobs, IS professionals are not necessarily more- or lesssusceptible to the negative impacts of RA and RC on job satisfaction. Also, the unexplained variance in the study effect sizes indicates the existence of potential moderating variables. To understand the various demographic and contextual factors that may differentiate various populations, future studies should examine other, theoretically meaningful moderators to explain the variations of both correlation coefficients across samples of IS employees Limitations It is important to clarify the limits or bounds of this study. First, because of the lack of individual reliability statistics from prior research [28], Hypothesis 4 is statistically tested based on the original correlations, instead of the weighted correlations after correcting for sampling error and measurement error. Second, due to the relatively small number of studies in the IS literature, compared with the minimum sample size required for moderating analysis, we were unable to conduct subgroup moderating analysis in our study Implications and Contributions This study is the first attempt in IS literature to synthesize empirical study results for the correlations of RA/RC with job satisfaction. The meta-analysis method allows us to take every available empirical study into consideration, overcoming the limitations of traditional narrative reviews and provides a quantitative measure of the relationship in question. Nevertheless, as advocated by Hwang [19], the purpose of meta-analysis is by no means to determine the final word, but to take stock and provide direction for future research (p. 44). Despite the limitations mentioned above, our research seeks to provide future research directions in the following areas Research Directions for RA/RC Area. Past IS studies have reported different levels of the relationship between RA/RC and job satisfaction. The fact that our meta-analysis supports both Hypotheses 1 and 2 in this study helps to reduce this confusion. The lack of support for Hypothesis 3 indicates that both RA and RC have similar levels of negative impact on job satisfaction. Thus, these two role stressors are both important predictors of IS professional s job satisfaction. This meta-analysis study also contributes to IS research by identifying the need to explore theoretically based moderating variables to account for the different levels of relationship between RA/RC and general job satisfaction. The large unexplained variance in the mean effect sizes is consistent with the two organizational behavior studies, which also advocated the need for identifying moderators for the relationship between RA/RC and other dependent variables (including selfrated and superior-rated job performance, tension/anxiety, and turnover intentions [10, 28]. Even though our study only examined one dependent variable (job satisfaction), considering the support for Hypotheses 4a and 4b (IS professionals are not necessarily less susceptible to the negative impacts of RA and RC on job satisfaction than other occupations), it is possible that the correlations between RA/RC with other outcome variables in IS setting are also sensitive to certain moderators. Thus, IS researchers should not limit their future research 7

8 regarding RA and RC to simple, bivariate relationships, but should examine possible moderators such as gender, job duties, self-efficacy, tolerance of ambiguity, need for structure and other personal attributes. Another future research direction is the investigation of the correlation between role stressors and the specific sub-dimensions of the job satisfaction scales (e.g., satisfaction with co-workers, pay, promotional opportunities, and the job task itself), since the organizational behavior meta-analyses [10, 28] showed that RA/RC differ in terms of their effect sizes with general job satisfaction versus other facets of job satisfaction Application of Meta-analysis in IS Field. The fourth issue is related to the broader application of metaanalysis method throughout the IS field than with the specific research area of RA and RC Misuse of the Term Meta-Analysis. It is not the focus of this research to provide a comprehensive methodological review of the use of meta-analysis in IS research, which has been covered quite thoroughly by Hwang [19]. Nonetheless, there is one problem that has emerged recently (from the mid-to-late 1990s) in IS research the misuse of the term meta-analysis. We used the terms meta-analysis or meta analysis as keywords to search for IS-related, published meta-analysis research in several online databases and conference proceedings (as describe in the Literature Search section) and located 21 studies. All 21 studies had meta-analysis or meta analysis in their article titles or abstracts, however nine 2 of these studies did not actually perform a true meta-analysis. Before we mention these studies, a restatement of the meta-analysis definition will help in clarifying the criterion used to judge whether a study constitutes a true meta-analysis. The term meta-analysis refers to the analysis of analyses the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings [11, p. 3]. This definition indicates that in order to be considered a metaanalysis study, the study should: 1) collect quantitative results from multiple prior studies; 2) perform statistical analysis on these quantitative data; and 3) focus on analysis of effect size, an indicator of the strength of the relationship between the relevant research variables [11]. The most common measures of effect sizes are Pearson s r (for survey research) and Cohen s d (for laboratory research). 2 Among the 21 studies, three of them are review papers for meta-analysis itself, nine of them didn t employ true metaanalysis procedures, the other nine of these studies (which are denoted with ** in the reference) did actually perform a true meta-analysis. The three criteria that we identified are used here to evaluate the nine papers that do not meet Glass s [11] definition of meta-analysis, defined above. Among the nine papers, two of them first qualitatively reviewed existing research to either develop a research framework [11] or to identify information system success (ISS) factors [36], and then collected first-hand data to either evaluate the framework or study the rank order of the ISS factors. Similarly, three other papers [15, 33, and 43] only conducted a qualitative review of prior research, with no quantitative data analysis involved. These five studies satisfied none of the three meta-analysis criteria. In all cases, the authors use of the term meta-analysis in their paper titles or abstracts actually referred to the traditional qualitative literature review which Hunter and Schmidt [18] label as narrative review. The other four papers met the first two criteria, in that they gathered quantitative data from prior research and performed some statistical analysis on these data. However, none of these studies considered effect sizes as the focus of their study. That is, none of them actually integrated statistical results from prior studies research variables. Instead, they examined other characteristics of the secondary data. Palvia and his colleagues [45] collected secondary data about the ranking of key IT issues, and then used cluster analysis to regroup them and used t-tests to verify the categories. Chau [6] investigated the relationship between construct reliability and research design characteristics using ANOVA to identify which design characteristics affect reliability. Esteves and Ramos [9] provide a descriptive statistical overview of IS studies conducted in Portugal. Kohli and Devaraj [34] employed logistic regression and discriminant analysis to examine the structural variables that affect IT payoff. While the last study seems closest to a true meta-analysis, the authors did not re-analyze the original effect sizes (correlation coefficients) from the original studies; rather, they simply counted the number of prior studies with positive and significant, zero, or non-significant findings as their form of analysis and conducted some analysis on the numbers of studies in each category. They did not examine the actual strength of the correlation, regression, or Lisrel paths in the prior studies and thus, their study neglected to analyze true effect sizes. These examples of studies that mis-use the term metaanalysis show that this method is not yet well understood by the IS research community. This is unfortunate, since there are meta-analyses in the IS literature going back over a dozen years [1]. An unexpected but noteworthy contribution of our study is to raise this issue and emphasize the correct use of the term, meta-analysis, which is elaborated in the following section Correct Use of the Term Meta-Analysis. The evaluation of the papers that do not meet Glass s [11] definition of meta-analysis indicates that the meaning of 8

9 meta-analysis has been diluted, so that some researchers tend to mis-label two other types of studies as metaanalysis. The first type is the traditional qualitative literature review or narrative review (e.g., [11], [36], [15], [33], and [43]). This is by no means intended to de-value the contributions that narrative reviews can make, but there are certain circumstances under which a qualitative review is unable to provide an accurate representation of population characteristics. One such situation is when the number of prior studies is large and the studies are not comparable in terms of sampling, measures, and so forth. This limitation can, to certain degree, be well handled by meta-analysis. For example, Hunter and Schmidt recommended procedures that can deal with variations in study effect size due to sampling error and other artifacts and with attenuation of effect-size estimates due to measurement unreliability and range restriction [18, p. 485]. Thus, qualitative literature review and meta-analysis are two different methods which are suited for different objectives the term meta-analysis is not a synonym for qualitative literature reviews. The second type of study that often being mis-labeled as meta-analysis is research that does include quantitative analyses based on existing literature, however, the quantitative analysis does not focused on prior studies research variables (e.g., [45], [6], [9], and [34]). Again, we have no intent to discredit the value of this type of study. Nevertheless, the emphasis of these studies is substantially different from the objective of metaanalysis. Therefore, the term meta-analysis is not interchangeable with this type of research. In conclusion, we advocate the term meta-analysis should only be used for those studies that: 1) collect quantitative results from multiple prior studies; 2) perform statistical analysis on these quantitative data; and 3) focus on analysis of effect size, an indicator of the strength of the relationship between the actual research variables from those prior studies[11]. 7. References 3 [1] Alavi, M. and Joachimsthaler, E. A. Revisiting DSS Implementation Research: A Meta-Analysis of the Literature and Suggestions for Researchers, MIS Quarterly, 16:1, March 1992, pp [2]*Baroudi, J.J. The Impact of Role Variables on IS personnel Work Attitudes and Intentions, MIS Quarterly, December 1985, pp [3] Bartol, K.M., and Martin, D. "Managing Information Systems Personnel: A Review of the Literature and 3 The papers denoted with * were included in our quantitative meta-analysis. The papers denoted with ** were the prior studies in the IS literature that did employ true meta-analysis procedures. Managerial Implications," MIS Quarterly (6), December 1982, pp [4]*Bostrom, R. P. "Role Conflict and Ambiguity: Critical Variables in the User-Designer Relationship," Proceedings of the Seventeenth Annual Computer Personnel Research Conference, Association for Computing Machinery Press, New York, 1981, pp [5]**Cao, Q., Nah, F.F., and Siau, K. A Meta-Analysis on Relationship Modeling Accuracy: Comparing Relational and Semantic Models, Proceedings of AMCIS Conference, 2000, Long Beach, CA, [6] Chau, P.Y.K. On the Use of Construct Reliability in MIS Research: A Meta-Analysis, Information & Management (35), 1999, pp [7] Cohen, T.D. Statistical Power Analysis for the Behavioral Sciences, (2nd ed.). Jillsdale, NJ: Lawrence Erlbaum Associates, [8]**Dennis, A.R., Haley, B.J., and Vandenberg, R.J. A Meta-Analysis of Effectiveness, Efficiency, and Participant Satisfaction in Group Support Systems Research, Proceedings of ICIS Conference, 1996, Cleveland, OH, [9] Esteves, J. and Ramos, I. A Meta-Analysis of Information Systems Research Diversity in Portugal, Proceedings of AMCIS Conference, 2003, Tampa, FL, [10] Fisher, C.D. and Gitelson, R. A Meta-Analysis of the Correlates of Role Conflict and Ambiguity, Journal of Applied Psychology (68:2), 1983, pp [11] Glass, G.V. Primary, secondary, and metaanalysis, Educational Researcher (5), 1976, pp [12]*Goldstein, D. K., and Rockart, J. F. "An Examination of Work-Related Correlates of Job Satisfaction in Programmer/Analysts," MIS Quarterly (8), 1984, pp [13] Greenhaus, J.H. and Beutell, N.J. Sources of Conflict Between Work and Family Roles, Academy of Management Review (10:1), 1985, pp [14]*Guimaraes, T. and Igbaria, M. Determinants of Turnover Intentions: Comparing IC and IS personnel, Information Systems Research (3:3), 1992, pp [15] Heine, M.L., Grover, V. and Malhotra, M. K. The Relationship between Technology and Performance: A Meta-Analysis of Technology Models, Omega (31:3), June 2003, pp [16] Hoy, F. Small Business Survey Questionnaire. Athens, GA: Small business Development Center Publications, The University of Georgia, [17] Huarng, A. S. Burnout syndrome among information system professionals, Information Systems Management (18:2), Spring 2001, pp. 15-xx. [18] Hunter, J.E. and Schmidt, F.L. Methods of Meta- Analysis, Correcting Error and Bias in Research Findings. Thousand Oaks: Sage Publication,

10 [19] Hwang, M.I. The Use of Meta-Analysis in MIS Research: Promises and Problems, The DATA BASE for Advances in Information Systems (27:3), 1996, pp [20]**Hwang, M. Did Task Type Matter in the Use of Decision Room GSS? A Critical Review and a Meta- Analysis, Omega (26: 1), 1998, pp [21]**Hwang, M.I. and Wu, B.J.P. The Effectiveness of Computer Graphics for Decision Support: A Meta- Analysis Integration of Research Findings, The DATA BASE for Advances in Information Systems (21:3), 1990, pp [22]**Hwang, M.I., Windsor, J.C., and Pryor, A. Building a Knowledge Base for MIS Research: A Meta- Analysis of a Systems Success Model, Information resources Management Journal (13:2), 2000, pp [23]*Igbaria, M. and Chidambaram, L. The Impact of Gender on career success of information systems professionals: A Human-Captial Perspective, Information Technology & People (10:1), 1997, pp [24] Igbaria, M., Meredith, G., and Smith, D.C., Predictors of Intention of IS Professionals to Stay With the Organization in South African, Information & Management (26), 1994, [25]*Igbaria, M., Parasuraman, S. and Badawy, M.K. Work Experiences, Job Involvement, and Quality of Work Life Among Information Systems Personnel, MIS Quarterly, June 1994, [26] Ivancevich, M., Napier, A. H., and Wetherbe, J. C. "Occupational Stress, Attitudes, and Health Problems in the Information Systems Professional," Communications of the ACM (26:10), 1983, pp [27]*Ivancevich, M., Napier, A. H., and Wetherbe, J. C. An Empirical Study of Occupatoinal Stress, Attitudes and Health Among Information Systems Personnel, Information & Management (9), 1985, pp [28] Jackson, S.E. and Schuler, R.S. A Meta-Analysis and Conceptual Critique of Research on Role Ambiguity and Role Conflict in Work Settings, Organizational Behavior and Human Decision Processes (36), 1985, pp [29] Jex, S. and Elacqua, T. Self-esteem as a Moderator: A Comparison of Global and Organization-based Measures, Journal of Occupational & Organizational Psychology (72:1), 1999, pp of Organizational & End User Computing (16:1), 2004, pp [42]**Mahmood, M.A., Hall, L., and Swanberg, D.L. Factors Affecting Information Technology Usage: A Meta-Analysis of the Empirical Literature, Journal of Organizational Computing and Electronic Commerce (11:2), 2001, pp [43] Marshall, C., Shipman, F. M., and McCall, R. J. Making Large-Scale Information Resources Serve [30] Kahn, R.L., Wolfe, D., Quinn, R., Snoek, J.D., and Rosenthal, R. Organizational Stress: Studies in Role Conflict and Role Ambiguity, John Wiley and Sons, New York, New York, [31] King, R.C. and Sethi, V., The Impact of Socialization on the Role Adjustment of Information Systems Professionals, Journal of Management Information Systems (14:4), Spring 1998, pp [32]*King, R.C. and Xia, W.D., Retaining IS Talents in the New Millennium: Effects of Socialization on IS Professionals Role Adjustment and Organizational Attachment, Proceedings of the ACM SIGCPR, San Diego, CA., 2001, [33] Klaus, H., Rosemann, M. and Gable, G.G. What is ERP? Information Systems Frontiers (2:2), 2000, pp [34] Kohli, R. and Devaraj, S., Measuring Information Technology Payoff: A Meta-Analysis of Structural Variables in Firm-Level Empirical Research, Information Systems Research (14:2), 2003, pp [35]*Lee, P.C.B., Turnover of Information Technology Professionals: a contextual model, Accounting, Management and Information Technology (10),, 2000, [36] Li, E.Y. Perceived Importance of Information System Success Factors: A Meta Analysis of Group Differences, Information & Management (32), 1997, pp [37]*Li, E.Y., and Shani, A. B. "Stress Dynamics of Information Systems Managers: A Contingency Model," Journal of Management Information Systems (7:4), 1991, pp [38]**Lim, J., Richardson, V.J., and Roberts, T.L. Information Technology Investment and Firm Performance: A Meta-Analysis, Proceedings of the 37th HICSS, 2004, HI, pp [39] Lipsey, M.W., & Wilson, D.B. (2001). Practical Meta-analysis. Thousand Oaks: Sage. [40] Locke, E. The Nature and causes of Job Satisfaction, in Handbook of Industrial and Organizational Psychology, M.D. Dunnette (ed.), Rand, Chicago, Illinois, 1976, pp [41]**Ma, Q. and Liu L. The Technology Acceptance Model: A Meta-Analysis of Empirical Findings, Journal Communities of Practice, Journal of Management Information Systems (11: 4), Spring 1995, pp [44] Nelson, A.C. and LeRouge, C., Self-Esteem: Moderator Between Role Stress Fit and Satisfaction and Commitment?, Proceedings of the ACM SIGCPR, 2001, San Diego, CA, pp [45] Palvia, P.C., Palvia, S.C.J., and Whitworth, J.E., Global Information Technology: A Meta-Analysis of Key Issues, Information & Management (39), 2002, pp