The Use of Meta-Analysis in Validating the Delone and McLean Information Systems Success Model

Transcription

1 The Use of Meta-Analysis in Validating the Delone and McLean Information Systems Success Model Mark Hwang The University of Texas-Pan American Edinburg, Texas Ephraim R. McLean Georgia State University Atlanta, Georgia Abstract The measurement of systems success is important to any research on information systems. A research study typically uses several dependent measures as surrogates for systems success. These dependent measures are generally treated as outcome variables which, as a group, vary as a function of certain independent variables. Recently, a process perspective has been advocated to measure system success. This process perspective recognizes that some of the dependent measures, along with independent variables, also have an impact on outcome variables. A comprehensive success model has been recently developed by DeLone and McLean [I] to group success measures cited in the literature into three categories: quality, use, and impact. This model further predicts that quality affects use, which in turn affects impact. This paper explores the plausibility of using metaanalysis to validate this success model. Advantages of using meta-analysis over other research methodologies in validating process models are examined. Potential problems with meta-analysis in validating this particular success model are discussed. A research plan that utilizes past empirical studies to validate this success model is described. The background of meta-analysis Coined and first applied by Glass [2], meta-analysis involves the use of statistical procedures to integrate research findings across studies. It has been warmly embraced in those fields in which experiments are widely used, such as medicine, education, and psychology (Cook [3]). MIS researchers have just begun to use metaanalysis to integrate research findings in areas such as DSS implementation (Alavi and Joachimsthaler [4], and GSS use (Benbasat and Lim [5], McLeod [6]). Although the techniques used may vary, all meta-analyses aim to derive a quantitative measure of the relationship under study - the effect size. Two common effect size measures are Cohen s d and Pearson s r. In addition to summarizing past research findings (e.g., whether or not GSSs are effective), meta-analysis can also be used to investigate causal relationships (e.g., why GSSs are effective). Causal explanation entails the analysis of two types of variables: moderator and mediator (Cook [3], Shadish and Sweeney [7]). While moderators have been examined in MIS meta-analyses (e.g., Benbasat and Lim [.5]), mediators have never been tested before. Being process-oriented, mediational models explain not only what causes an effect, but also how the effect takes place. This type of knowledge is especially useful to practitioners because it is readily transferable (Cook [3]). Testing mediational models in meta-analysis Procedure The procedure for testing mediational models in me&analysis is the same as that in individual studies except that, in meta-analysis, the test data come from empirical studies conducted in the past. Mediational models are usually investigated in meta-analysis via path or structural modeling techniques. For example, Shadish and Sweeney [7] used EQS (Bentler [8]) to investigate the controversial effect of theoretical orientation to psychotherapy on marital and family psychotherapies. They hypothesized that theoretical orientation had an effect on therapy outcomes, but that the effect was mediated by four variables specific to a therapy or a research. The fitted model confirmed that orientation affects mediators, which in turn affect therapy outcomes. Traditional path modeling techniques and LISREL have also been used to test mediational models in meta-analysis (e.g., Brown and Peterson [9], Premack and Hunter [lo], Rothstein [ 111) /96 $ IEEE 176

2 Advantages While testing mediational models is possible in individual studies, the endeavor is more fruitful in metaanalysis. One reason is that meta-analysis permits the testing of hypotheses that were not built into individual studies (Hale and Dillard [ 121). In group support systems research, for example, a mediational model is generally adopted which posits that technology affects group processes, which in turn affect group outcomes (e.g., Dennis et al. [13]). GSS researchers have measured a variety of group process and outcome variables. However, with rare exceptions (e.g., Gopal, Bostrom, and Chin [14]), process variables were treated no different from outcome variables in previous studies. For example, participation and consensus have been measured in a number of studies as dependent variables (e.g., Gallupe, DeSanctis, and Dickson [15]; Ho and Raman [16];, Ho, Raman, and Watson [17]). However, none of these studies treated participation as a mediational variable, as suggested in GSS conceptualizations (e.g., Dennis et al. [ 131). A mediational model can be validated in a metaanalysis by fitting it to effect sizes calculated on participation and consensus from individual studies. Another advantage of testing mediational models in meta-analysis is that the relationship has a better chance of being correctly understood due to the potentially higher general validity of meta-analysis (Cook [3]). As argued by Cook [3], meta-analysis can enhance the general validity of conclusions through an increase of the four types of validity mentioned by Cook and Campbell [ 181: statistical conclusion validity, internal validity, construct validity, and external validity. Meta-analysis can enhance statistical conclusion validity because highly reliable mean differences are used as the unit of analysis, which may outweigh any loss in statistical power due to the sample of studies being smaller than that of individual studies. Internal validity can be enhanced because sources of bias (e.g., poorly designed research) in individual studies may cancel out or be tested for empirically. Because the reliability of a sample value affects statistical power (Cohen [19]), the reduction in bias in a meta-analysis, in turn, enhances its statistical conclusion validity. Construct validity of causes and effects can be enhanced because many realizations of a treatment and effect are tested in a metaanalysis. Finally, external validity can be enhanced because meta-analysis allows for testing of causal connections in a wider range of persons, settings, and times than is possible in individual studies. Potential problems Despite their promise, mediational models have not been studied extensively in me&analysis (Shadish and Sweeney [7]). The paucity of such endeavors may be due to the low availability of data about relevant constructs and the difficulty associated with ruling out alternative models (Cook [3]). The first problem stems from the fact that meta-analysis relies on data from prior studies. Because different studies use different variables, there may not be enough data about a construct of interest in a meta-analysis to estimate its path coefficients. As a result, mediational models tested in a meta-analysis would be simplified versions of theoretical models. For instance, in addition to participation, Dennis et al, [15] identified six process variables that may impact group outcomes. However, except for participation, none of these process variables were included in previous meta-analyses (Benbasat and Lim [5], McLeod [6]) due to a lack of data. Any meta-analysis in the future that tests mediational models of GSS will likely exclude those variables for the same reason. The second problem, which is not unique to metaanalysis, is relevant whether mediational models are tested in individual studies or review studies such as me&analysis. To counter this problem, a meta-analysis should only test mediational models that are theory based. Ruling out alternative models is becoming less of a problem as theories are advanced to guide research, as exemplified in GSS research (e.g., Poole and DeSanctis [20], Rao and Jarvenpaa [21]). Testing the Delone and McLean I/S success model in a meta-analysis This section describes a research plan that could validate the information systems success model described by DeLone and McLean [l]. The advantages and disadvantages of using meta-analysis for this endeavor are discussed first, followed by a description of the procedure that will be used. Advantages The success model of DeLone and McLean [l] includes six success variables (see Figure 1). Of the 100 studies that were used to develop this model, none measured all six variables; in fact, only 28 studies measured more than one variable. Apparently, measuring 177

3 System Quality Information Quality Figure 1. I/S success model. Source: DeLone and McLean (1992) multiple variables in a single study is not a common practice and any attempt to measure all six variables and test their interactions will likely be rare. Even if individual study could be carried out to test the success model specifically, their conclusions will not be as robust as those obtained through meta-analysis. Being an integration of past research findings, a meta-analysis conclusion is enhanced through an increase of the four types of validity of Cook and Campbell [18], as discussed previously. Potential Problems To test a path model that includes six variables, 15 intercorrelations must be available. This could be a problem because researchers generally focus on the effects of independent variables on dependent variables. Correlations among dependent variables (success variables) are either not reported or not measured at all. Of the 28 studies that provided multiple measures, there may or may not be enough data to calculate all 15 interconelations. Consequently, some of the paths in the success model may not be testable. Ruling out alternative models is less of a problem because the success model is theory based. Specifically, the model is based on the process perspective of organizational effectiveness of Steers [22] and on the ecology model of organizational effectiveness of Miles [23]. Procedure This section describes alternative data analysis methods that may be used to test the success model. First, the 28 studies that measured multiple variables can be reviewed to obtain intercorrelations among the six success variables. Their authors can also be contacted to see if such data are available. Depending upon available data, the final success model tested may or may not be the same as the one discussed by DeLone and McLean [ 11. Nevertheless, any relations tested will contribute to the understanding of IS success. An alternative to obtaining intercorrelations from individual studies is to calculate intercorrelations from among the measures reported in these individual studies. Using this approach, an effect size such as Pearson s correlation is calculated for every measure used in every study. Effect sizes calculated for the same measure from individual studies form a sampling distribution of this measure. Intercorrelations among these sampling distributions can then be calculated and used to test the success model. A problem with the second approach is that the six sampling distributions will have a lot of missing values because all studies measured only a portion of the six success variables. Pairwise deletions will render some intercorrelations incalculable. For instance, according to Table 8 of DeLone and McLean [l], none of the studies measured both information quality and organizational impact. Thus, the intercorrelation of these two measures is incalculable. Some pairs of measures were only available from a single study (e.g., only Miller and Doyle [24], measured both system quality and organizational impact) and, therefore, this intercorrelation is incalculable, also. In fact, most pairs of measures have very small number of values because they have not been consistently used. To remedy this problem, those missing values can be substituted with some hypothetical values, such as the mean of the sample. This is justified if the mean effect size is representative of effect sizes found in individual studies. The answer to the question whether the mean effect size is representative of individual effect sizes de- 178

4 pends upon the population variance: a small population variance means yes, a large population variance means no. When the population variation is large, therefore, the sample needs to be further analyzed to see if certain moderators may have caused wide variations among individual effect sizes. Hunter and Schmidt [25] propose a subgroup analysis to analyze moderators. Using this approach, a sample of effect sizes is broken into subgroups based on potential moderators. A true moderator will cause the variance in both subgroups to decrease. Assuming that no other moderators exist, the mean effect size of both subgroups is representative of effect sizes found in their respective groups. The subgroup mean effect sizes can then be used to substitute for missing effect sizes for individual studies which belong to the same group based on the moderator examined. Once the intercorrelations are calculated, a correlation matrix can be input to any structural modeling program to obtain path coefficients. Models rarely fit on the first test and several iterations may have to be performed before a satisfactory model emerges. Regardless of the result, a better understanding of IS success should be achieved in the end. Limitation of the study An often-cited problem in meta-analysis is the apples and oranges problem - the mixing of studies with different measuring techniques, definitions of variables, and design quality. These are valid concerns; however, any review, be it qualitative or quantitative in nature, entails some mixing of apples with oranges because pure replications are rare in any research domain. The success model of Delone and McLean [l], for example, was derived from studies with different measuring techniques, definitions of variables, and design quality. The fact that diverse studies were included in a review strengthens, not weakens, its conclusions (Kraemer and Pinsonneaulty [26]). As discussed previously, a heterogeneous sample in a meta-analysis also strengthens its conclusions with enhanced construct and external validity. In sum, heterogeneity poses a potential threat to the internal validity of a meta-analysis; at the same time, it provides an opportunity for enhanced construct and external validity. The bottom line is, therefore, can the threat to internal validity be ruled out? There is no easy answer to the above question in a qualitative review. Happily for meta-analysis, the answer can be derived empirically by testing the moderating effects of methodological differences. For example, Benbasat and Lim [5] tested year of publication as a possible bias in their meta-analysis and concluded that year of publication did not have an effect on the effectiveness of GSS. In this case, the threat to internal validity due to year of publication can be ruled out. The current meta-analysis will first validate the success model irrespective of any methodological differences inherent in the studies reviewed. A validated model in this case has great construct and external validity. If the model cannot be completely validated, additional tests can then be conducted on subgroups of studies based on the classification of a potential moderator variable. For example, validation can be performed separately on GSS studies and on DSS studies. Differences in the results, if any, can then be attributed to the different technologies. The process of model validation and any models validated as a result will both contribute to the understanding of IS success. Conclusions In this brief review, meta-analysis is proposed as a viable alternate to the replication of individual studies, in this case, studies dealing with information systems success. Such replication is often very costly and, in some cases, may not even be possible. Through metaanalysis, the validity of individual studies can be strengthened in four ways: construct, internal, external, and statistical conclusion validity. The Delone and McLean information systems success model, reflecting nearly a hundred individual studies, is suggested as a likely candidate for the application of meta-analysis. With its six dimensions of system success - system quality, information quality, system use, user satisfaction, individual impact, and organizational impact - the model provides a rich test bed for the use of meta-analysis to confirm - or refute - its structural and internal relationships. The conduct of such analysis is the next step. References DeLone, W.H., and McLean, E.R. Information Systems Success: The Quest for the Dependent Variable, Information Systems Research, 3(l), 1992, Glass, G.V. Primary, Secondary, and Meta-Analysis of Research, Educational Researcher, 5(10), 1976, 3-8. Cook, T.D. Meta-Analysis: Its Potential for Causal Description and Causal Explanation within Program Evaluation, in Social Prevention and the Social Sciences: Theoretical Controversies, Research Problems, and Evaluation Strategies, G. Albrecht and H. 179

5 Otto (Eds.), Walter de Gruyter, Berlin, 1991, Alavi, M., and Joachimsthaler, E.A. Revisiting DSS Implementation Research: A Meta-Analysis of the Literature and Suggestions for Researchers, MIS Quarterly, 16(l), March 1992, Benbasat, I., and Lim, L. The Effects of Group, Task, Context, and Technology Variables on the Usefulness of Group Support Systems: A Meta- Analysis of Experimental Studies, Small Group Research, November 1993,24(4), McLeod, P.L. An Assessment of the Experimental Literature on Electronic Support of Group Work: Results of A Meta-Analysis, Human-Computer Znteraction, (7) 1992, Shadish, W.R., and Sweeney, R.B. Mediators and Moderators in Meta-Analysis: There s a Reason We Don t Let Dodo Birds Tell Us Which Psychotberapies Should Have Prizes, Journal of Consulting and Clinical Psychology, 59(6), 1991, Bentler, P.M. EQS: A Structural Equations Program Manual, BMDP Statistical Software, Los Angeles, CA, Brown, S.P., and Peterson, R.A. Antecedents and Consequences of Salesperson Job Satisfaction: Meta- Analysis and Assessment of Causal Effects, Journal of Marketing Research, (30), February 1993, Premack, S.L., and Hunter, J.E. Individual Unionization Decisions, Psychological Bulletin, 103(2), 1988, Rothstein, H.R. Meta-Analysis and Construct Validity, Human Pedormance, 5(1&Z), 1992, Hale, J.L., and Dillard, J.P. The Uses of Meta- Analysis: Making Knowledge Claims and Setting Research Agendas, Communication Monographs, (58), December 1991, Dennis, A.R., George, J.F., Jessup, L.M., Nunamaker, J.F. Jr., and Vogel, D.R. Information Technology to Support Group Work, MIS Quarterly, 12(4), 1988, pp Gopal, A. Bostrom, R.P. and Chin, W.W. Applying Adaptive Structuration Theory to Investigate the Process of Group Support Systems Use, Journal of Management Information Systems, 9(3), 1993, Gallupe, R.B., DeSanctis, G., and Dickson, G.W. Computer-Based Support for Group Problem Finding: An Experimental Investigation, MIS Quarterly, 1988, 12(2), Ho, T.H., and Raman, KS. The Effect of GSS and Elected Leadership on Small Group Meetings, Journal of Management Information Systems, 1991, 8(2), Ho, T.H., Raman, K.S., and Watson, R.T. Group Decision Support Systems: The Cultural Factor, Proceedings of the Tenth International Conference on Information Systems, Boston, MA, 1989, Cook, T.D., and Campbell, D.T. Quasi-Experimentation: Design and Analysis Issues for Field Settings, Rand McNally, Chicago, Cohen, J. Statistical Power Analysis for the Behavioral Sciences, (2nd ed.), Lawrence Erlbaum Associates, Hillsdale, NJ, Poole, M.S., and DeSanctis, G. Understanding the Use of Group Decision Support Systems: The Theory of Adaptive Structuration, in Theoretical Approaches to Information Technologies in Organizations, C. Steinfeld and J. Fulk (Eds.), Sage Publications, Beverly Hills, CA, 1989, Rao, V.S., and Jarvenpaa, S.L. Computer Support of Groups: Theory-Based Models for GSS Research, Management Science, 1991,37(10), Steers, R.M. When Is an Organization Effective: A Process Approach to Understanding Effectiveness, Organizational Dynamics, (5) Autumn 1976, Miles, R.H. Macro Organizational Behavior, Good- year, Santa Monica, CA, Miller, J., and Doyle, B.A. Measuring Effectiveness of Computer Based Information Systems in the Financial Services Sector, MIS Quarterly, 1 l(1) March 1987, Hunter, J.E., and Schmidt, F.L. Methods of Meta- Analysis: Correcting Error and Bias in Research Findings, Sage Publication, Beverly Hills, CA, Kraemer, K.L., and Pinsonneault, A. The Implications of Group Support Technologies: An Evaluation of the Empirical Research, Proceedings of the 23rd Hawaii International Conference on Systems Sciences, (Vol. 3), IEEE Computer Society Press, 1990,