Examining context effects in organization survey data using IRT

Transcription

1 Rivers, D., Meade, A. W., & Fuller, W. L. (2007, April). Examining context effects in organization survey data using IRT. Paper presented at the 22nd Annual Meeting of the Society for Industrial and Organizational Psychology, New York. Examining context effects in organization survey data using IRT Drew Rivers and Adam W. Meade North Carolina State University W. Lou Fuller Duke Energy Organizational researchers often modify employee surveys over time. However, changes to the survey form can introduce measurement artifacts, such as context effects, leading to differences in observed responses that are not due to true organizational change. This paper illustrates the use of IRT to identify context effects in organizational surveys. Surveys are perhaps the most frequently applied data collection technique used by organizations (Church & Waclawski, 2001; Kraut & Saari, 1999; Sackett & Larson, 1990; Stanton, Sinar, Balzer, & Smith, 2002). One reason for their widespread use is the enormous amount of flexibility in the variety of purposes that they can serve (Church, 2001). For example, organization surveys are commonly used to identify areas of concern, analyze long-term trends, evaluate the impact of organizational change, provide information for future decisions, analyze organizational behavior, serve as a communication channel, improve organizational functioning, and to provide symbolic communication (Kraut, 1996). Additionally, surveys are relatively nonintrusive and inexpensive compared to other data collection procedures (Church, 2001). Evidence of the importance organizations place on surveys can be witnessed by the existence of consortia dedicated to providing benchmarks for survey data across organizations and industries. Further, Kraut (1996) estimates that more than half of US companies regularly survey their employees. The frequent application of organizational survey data in the literature is a testament to its continued value to both academics and practitioners alike. Despite the widespread use of organizational surveys in both research and practice, relatively little attention has been given to the potential presence of extraneous factors that can affect survey data. While common in other areas of research (e.g., public opinion surveys; Colasanto, Singer, & Rogers, 1992), context effects (to which they are commonly referred) have rarely been discussed in organizational research. Context effects occur when responses to questionnaire or interview items vary as a function of other items on the instrument. Contexts effects can have a strong influence on survey responses, particularly for common attitudinal measures such as job satisfaction and organizational commitment, yet can be difficult to detect by examining item content or basic item and scale statistics alone. Undetected, context effects can strongly mislead persons relying on survey results as an accurate assessment of organizational phenomena (e.g., culture, satisfaction). In this study, we propose the use of item response theory (IRT) as a means to detect context effects in longitudinal organizational survey research. Longitudinal Organizational Surveys and Context Effects The analysis of organizational survey data across time offers benefits both to academics, in terms of developing and testing theories, and to practitioners, with regard to designing and evaluating business initiatives. However, organizational survey programs are under constant pressure to change survey content, item wording, and scale formats to meet organizational needs (Higgs & Ashworth, 1998). While such changes are necessary to meet the evolving needs of the organization, these changes also introduce the potential for context effects, leaving the researcher unsure of whether changes in responses across time are attributable to real changes or to artifacts associated with the different survey versions. Context effects occur when responses to questionnaire (or interview) items differ based on the presence of other items on the survey. Underlying this phenomenon is the idea that attitude judgments are temporary and malleable constructions, rather than fixed or stable evaluations. Tourangeau and Rasinski (1988) describe attitudes as structures in long-term memory comprised of interrelated beliefs. These beliefs encompass experiences, feelings, general propositions, knowledge, past judgments, and

2 Context Effects & IRT 2 images regarding a particular attitude object or target stimulus. Importantly, while the attitude structure may remain stable in memory, an individual may report different attitudinal judgments depending on the context in which the attitude is elicited. While attitudinal judgments are formed based on information stored in memory, not all relevant information is retrieved when prompted for an attitude judgment. Individuals will generally truncate the retrieval process once enough information has been accessed to form a judgment (Schwarz, 1995). Further, the accessibility of information can be either chronic or temporal (Schwarz, 1995; Schwarz & Bless, 1992). Generally, chronically accessibility information is used frequently, and is therefore readily accessible and independent of contextual cues. Conversely, temporarily accessible information is used infrequently, but may be primed into retrieval by contextual cues, for example, earlier items in a questionnaire. The assumption of information accessibility underlies two models of context effects: the beliefsampling model (Tourangeau, 1999; Tourangeau, Rips, & Rasinski, 2000) and the inclusion/exclusion model (Schwarz & Bless, 1992; Schwarz, 1995). According to the belief-sampling model, respondents utilize a sample of all relevant beliefs stored in memory when forming a judgment, and the judgment is based on an aggregation of these beliefs (Tourangeau, 1999). Since the context can influence what beliefs are sampled, the reliability of attitudinal judgments across situations depends on three factors: (1) The homogeneity of the underlying pool of relevant beliefs, (2) the consistency in the sampling of beliefs, and (3) the consistency in the aggregation of beliefs to form a judgment. The inclusion/exclusion model also rests on the assumption of information accessibility. Schwarz and Bless (1992) suggest that in order for individuals to form a judgment about a target stimulus, a representation of both the target and a standard of comparison must be constructed. Both constructions are, in part, dependent on the context in which the judgment is elicited. Again, not all relevant information is retrieved to make the judgment. Information made temporarily accessible may be included in the representation of the target, or it may be excluded from the target and subsequently incorporated into the representation of the standard. The inclusion or exclusion of contextual information depends on the perceived appropriateness and relevance to the target stimulus (Schwarz, 1995). When information is included into the target representation, the effect is referred as assimilation; when it is excluded from the target and included into the standard, it is referred to as contrast (i.e., exclusion). In general, target questions that allow for the inclusion of a wide range of information are more likely to generate an assimilation effect. Conversely, specific target questions are more likely to elicit a contrast effect. A study by Colasanto et al. (1992) offers an example of how context can influence target item interpretation. The authors suspected a context effect to be partially responsible for a significant change in public opinion about contracting AIDS by donating blood: opinions jumped from 28.9% to 43.5% in a one-year period. In both opinion polls, respondents were asked about nine different methods through which a person might contract AIDS, including donating blood and receiving a blood transfusion. In the earlier opinion poll, the donating blood method always followed the blood transfusion method. In the later poll, the methods of transmission were randomly presented across respondents. To test for a context effect, Colasanto and colleagues conducted a split-ballot experiment, with half the sample asked about blood transfusion first, and the other half about donating blood first. The researchers found that those asked about blood transfusion first were less likely to believe AIDS could be acquired by donating blood. In this case, respondents may have excluded information about blood transfusions from their representation of the target item, blood donations. Studies that involve naturally occurring context effects tend to be an exception in the literature. As Tourangeau, Singer, and Presser (2003) note, much of the work in this area tends to involve controlled experiments designed to generate context effects. Organizational researchers, however, often have little control over survey design and administration. As a result, carefully controlled experimental design and sampling is not possible. Fortunately, item response theory (IRT) provides a method, superior in many ways to experimental mean comparisons, that can be used to evaluate the presence of context effects in organizational surveys. IRT & Differential Item Functioning IRT models the relationship between the probability of observed response and an examinee s level of some latent trait or attitude. Perhaps the most commonly used IRT model in organizational research has been Samejima s (1969) graded response model (GRM). In the GRM, a series of boundary response functions (BRFs) are graphed that relate the probability of responding at or above a given response category to the respondent s underlying level of the latent trait or attitude (such as satisfaction). A set of parameters is estimated for each item in the GRM, which is then used to plot the

3 Context Effects & IRT 3 BRFs. These parameters can also be used to plot category response curves (CRCs). The CRCs represent the probability of responding in a given response category across the range of the latent trait. We refer the reader to Embretson and Reise (2000) for a much more thorough discussion of the GRM. When multiple samples (e.g., groups, time periods, response formats, etc.) respond to the same set of items, item parameters can be estimated separately for those samples. The extent to which the item functions differently for the groups is commonly referred to as differential item functioning (DIF), though the term item parameter drift is sometimes used in DIF studies of longitudinal data (e.g., Donoghue & Isham, 1998; Wells, Subkoviak, & Serlin, 2002). Conceptually, DIF in IRT is highly similar to measurement invariance research using confirmatory factor analysis (Maurer, Raju, & Collins, 1998; Meade & Lautenschlager, 2004; Raju, Laffitte, & Byrne, 2002). Raju and colleagues (Flowers, Oshima, & Raju, 1999; Raju, van der Linden, & Fleer, 1995) have developed the differential functioning of items and tests (DFIT) framework to identify practically significant DIF at the item and test level. A technical presentation of DFIT is available in the aforementioned articles, but note that DFIT has been used many times in organizational research (e.g., Facteau & Craig, 2001; Maurer et al., 1998). Study Overview This study will examine organization survey data from a large US-based energy company. The Company has maintained a survey program for nearly two decades and has transitioned the survey through various modes of data collection and numerous organization changes. The survey program adheres to standards of survey research established by a national consortium of top US firms, of which the Company has been a member for more than a decade. While the Company has attempted to minimize changes to the survey content, such changes are inevitable in a dynamic and evolving workplace. In 2004, the Company incorporated a series of changes to the survey, which was associated with a significant decline in favorable ratings to an item: How satisfied are you with the information you receive from management on what's going on at this company? (Hereafter referred to as the target item.) Historically, the target item received favorable ratings (defined as ratings of four and five on the five point scale) in the range of 49% to 53% (See Figure 1). Following the 2004 administration favorable ratings dropped to 33%, a 16 percentage point decrease from the previous year. No significant, negative organization events occurred in the interim that could explain the precipitous decline. However, the Company s survey administrator suspected a potential context effect resulting from the inclusion of a new item immediately preceding the target item. In order to investigate the possibility of a context effect, we employed IRT methods of DIF assessment. If changes to the survey form altered the interpretation of the target item, the DIF analysis should reveal a difference in item functioning from 2003 to Method Survey Instrument This study focuses on a subset of items that address employee attitudes toward the Company s direction and general communications with its employees. These items were scattered throughout the original survey form, but factor analysis revealed that a single factor (Goals and Direction) underlay these items. The items comprising this factor appear in Table 1. Item 10 in the scale is the target item and was the last of these items to appear in the survey. Participants The participants include employees of the Company for the years 2003 and 2004, who: (1) were invited to participate in the survey, (2) responded to all Goals and Direction items, and (3) reported their primary work location within the USA (to control for potential DIF due to translation and cross-cultural differences; Ellis, Minsel, & Becker, 1989). The final dataset included N=10,440 respondents in 2003 and N=6,651 respondents in Table 2 depicts descriptive statistics for the Goals and Direction items. Note that with IRT, it is not necessary that the exact same respondents are included in the 2003 and 2004 samples, as the relationship between item response and latent trait does not vary, even when mean differences in latent traits exist. Procedure Unidimensionality. An important assumption underlying IRT models is unidimensionality (Hambleton, Swaminathan & Rogers, 1991). Exploratory factor analysis with principal axis factoring was conducted to assess the dimensionality of the 10-item scale using the combined 2003 & 2004 sample. IRT Analyses. Item parameters were estimated separately in 2003 and 2004 using the GRM and MULTILOG 7.03 (Thissen, 1991). One requirement of IRT DIF procedures is that item parameter estimates be put onto the same metric. We used the EQUATE 2.1 program (Baker, 1995; cf. Raju et al., 1995), and then used these linked item

4 Context Effects & IRT 4 parameters as input into the DFITPS6 program (Raju, 1999). Since equating assumes the common anchor items are invariant between groups, we used an iterative linking process to identify a stable set of anchor items (see Lautenschlager, Flaherty, & Park, 1994 and Park & Lautenschlager, 1990 for a discussion of the advantages of iterative linking). Results The EFA scree plot indicated a unidimensional scale (see Figure 2) with the first factor explaining 51.7% of item variance. In addition, confirmatory factor analysis using Mplus 3.13 (Muthen & Muthen, 2005) was conducted to test whether a single factor model could adequately explain the data. Again, the samples were combined for the analysis. Results suggest an adequate fit: CFI =.935, TLI =.916; SRMR=.042. Overall, the unidimensionality of the Goals and Direction scale was upheld. The iterative linking process revealed the presence of only one DIF item at both the first and second stage of linking. Table 3 shows DFIT results following the final stage of linking. An item was considered differentially functioning when the NCDIF index exceeded the.096 cut-off value recommended by Raju (1999). Only the target item, Item 10, showed DIF while the overall scale did not show differential test functioning (DTF=.143 versus the recommended cutoff,.960). To further explore Item 10, BRFs and CRCs were generated for the 2003 and 2004 samples using the equated item parameters. Equated parameters for Item 10 are presented in Table 4. Figure 3 depicts the BRFs for Item 10 while CRCs offer another view of the DIF between samples (see Figure 4). The interpretation with both the BRFs and CRCs is that a member of the 2003 sample has a higher probability of selecting a response of 4 or 5 on the scale than a member of the 2004 sample given the same level of satisfaction. Put differently, members of the 2004 sample required higher levels of satisfaction to have the same probability of responding with satisfied or very satisfied. This finding is consistent with the decline in favorable ratings uncovered by the Company in 2004 and indicates that this decline is unlikely to be due to a true drop in satisfaction. Rather, the item functions differently on the 2003 and 2004 surveys, thus no change in the overall sample latent satisfaction would be necessary to witness a lower observed mean on the 2004 survey for Item 10. Discussion Observed item score differences over time can emerge for one of two reasons. First, the item could accurately measure the trait or attitude on which respondents actually differ over time, or second, an item functions differently over time leading the an inaccurate perception of change. Item 10 in the Goals and Direction scale appears to have functioned differently in 2003 and 2004, likely due to a survey context effect. This context effect then erroneously leads to an impression of lower satisfaction over time. These findings have several implications. For researchers, this study is the first to illustrate the use of IRT for identifying context effects with organizational survey data. The IRT methods illustrated here are more advantageous than traditional split-ballot designs for several reasons. First, unlike traditional techniques, the IRT method does not necessitate the administration of multiple survey forms, which is seldom feasible with largescale organizational surveys. Second traditional methods are prone to problems associated with the unreliability of single-item measures (the outcome measure in split-ballot designs) whereas the IRT model accommodates for unreliability (see Embretson & Reise, 2000). Third, IRT accommodates for differences in the latent trait or attitude over time, eliminating the need for carefully matched samples across time periods. For the Company, these results have implications for organizational decision makers. Item 10 is an invalid indicator of changes across time with regard to employee satisfaction with information from management. Actions based mainly on the item alone have limited support in the data. Alternatively, other scale items and the scale itself were shown to lack DIF. An assessment of the survey context surrounding the target item reveals several changes in the survey form that may have generated a context effect. Table 5 displays the context for the target item in 2003 and Table 6 does the same for In order to better understand the potential source of the context effect, we examined two significant changes to the survey between 2003 and The first was a change in the item immediately preceding the target item (see Context Item 2 in Tables 5 and 6). In the 2003 survey form, this context item asked employees to select their primary information source regarding general news about the company. The method of response required the selection of one information source among a list of six alternatives. In the 2004 survey form, this item was replaced by a Likert scaled item asking employees to rate their agreement with a statement

5 Context Effects & IRT 5 regarding the credibility of company communications to employees. According to the inclusion/exclusion model (Schwarz & Bless, 1992), respondents construct a representation of the target item and of a standard of comparison prior to making a judgment. These constructions are influenced in part by the context in which the item is presented. In the present case, the change in the preceding item could have altered the representation of the target item. Specifically, in 2003 the preceding item could have influenced employees to interpret the target item as addressing satisfaction with the quantity or availability of information from management. In 2004, the preceding item could have swayed the interpretation of the target toward the quality or honesty of communication from management. In both cases, the effect was one of assimilation, whereby information from the context is integrated into the representation of the target. The belief sample model (Tourangeau et al., 2000) offers a similar explanation. According to the model, respondents will retrieve only a sample of relevant information to form a judgment about the target item: in this case, satisfaction with information from management. If the attitude judgment involves an aggregation of beliefs across various dimensions for example, information availability, information quality, information honesty, information timeliness, and information quantity then judgments are more likely to vary across different contexts. In this case, the preceding item in 2003 may have made beliefs about information availability or quantity more accessible and therefore more prominent when forming a judgment on the target item. Conversely, in 2004 the preceding may item have made beliefs about information quality or honesty more available. The second significant change made on the 2004 survey form suggests a context effect may have occurred at the final stage of the response process: mapping the judgment onto the scale (Tourangeau & Rasinski, 1988). The 2003 target item utilized a Likert scale ranging from Very Satisfied to Very Dissatisfied. In 2004, the target item incorporate a different set of anchors, ranging from To a Very Great Extent down to To a Little or No Extent. While the 2003 anchor descriptions ranged from positive to negative, the 2004 anchors effectively included a zero point (i.e., To a Little or No Extent) at one end of the scale. A mapping of 2003 anchors on to the 2004 anchors would likely shift favorable judgments in 2003 (e.g., Satisfied) down to neutral judgments in 2004 (To Some Extent). Examination of the Category Response Curves (Figure 4) offers some support for this notion. Ultimately, the source of the context effect likely emerged from several different changes to the survey form, including changes to both the preceding item and the scale anchors. In any case IRT proved a useful tool for identifying the presence of a context effect. Limitations Despite the apparent support for the context effect hypothesis, some limitations are worth noting. From a theoretical perspective, the relationship between context effects and DIF remains unclear. If context effects can occur at different phases of the response process (Tourangeau, 1999), at which phase will context effects produce DIF? For example, two groups may generate a similar interpretation of an item yet use a different standard of comparison when forming a judgment. Clearly further research is needed on the role of IRT to detect context effects in organizational survey research. Research designs like the Colosanto et al. (1998) that incorporate DIF analysis may help further refine this application of IRT. References Baker, F. B. (1995). EQUATE 2.1. A Fortran-based computer program used for equate item parameters between two groups. University of Wisconsin. Church, A. H. (2001). Is there a method to our madness? The impact of data collection methodology on organizational survey results. Personnel Psychology, 54, Church, A. H., Waclawski, J. (2001). Designing and using organizational surveys: A seven step process. San Francisco: Jossey-Bass. Colsanto, D., Singer, E. & Rogers, T. (1992) Context effects on responses to questions about AIDS. Public Opinion Quarterly, 56, Donoghue, J. R., & Isham, S. P. (1998). A comparison of procedures to detect item parameter drift. Applied Psychological Measurement, 22, Ellis, B. B., Minsel, B., & Becker, P. (1989). Evaluation of attitude survey translations: An investigation using item response theory. International Journal of Psychology, 24, Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum Associates. Facteau, J. D., & Craig, S. B. (2001). Are performance appraisal ratings from different rating sources comparable? Journal of Applied Psychology, 86,

6 Context Effects & IRT 6 Flowers, C. P., Oshima, T.C., & Raju, N. S. (1999). A description and demonstration of the polytomous-dfit framework. Applied Psychological Measurement, 23, Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of Item Response theory. Newbury Park, CA: Sage Publications. Higgs, A. C., & Ashworth, S. D. (1998). Organizational surveys: Tools for assessment and research. In A. I. Kraut (Ed.). Organizational Surveys: Tools for Assessment and Change. San Francisco, CA: Jossey-Bass. Kraut, A. I. (1996). An overview of organizational surveys. In A. I. Kraut (Ed.), Organizational surveys: Tools for assessment and research (pp. 1-14). San Francisco, CA: Jossey-Bass. Kraut, A. I., & Saari, L. M. (1999). Organizational surveys: Coming of age for a new era. In A. I. Kraut & A. K. Korman (Eds.), Evolving practices in human resource management: Responses to a changing world of work (pp ). San Francisco: Jossey-Bass. Lautenschlager, G. J., Flaherty, V. L., & Park, D. G. (1994). IRT differential item functioning: An examination of ability scale purifications. Educational and Psychological Measurement, 54, Maurer, T. J., Raju, N. S., & Collins, W. C. (1998). Peer and subordinate performance appraisal measurement equivalence. Journal of Applied Psychology, 83, Meade, A. W., & Lautenschlager, G. J. (2004). A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods, 7, Muthen, B., & Muthen, L. (2004). Mplus A computer program for structural modeling. Los Angeles, CA. Park, D. G., & Lautenschlager, G. J. (1990). Improving IRT item bias detection with iterative linking and ability scale purification. Applied Psychological Measurement, 14, Raju, N. (1999). DFITPS6. A Fortran-based computer program for calculating DIF/DTF. [Computer Program] Raju, N. S., Laffitte, L. J., & Byrne, B. M. (2002). Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology, 87, Raju, N. S., van der Linden,W. J., & Fleer, P. F. (1995). IRT-based internal measure of differential functioning of items and tests. Applied Psychological Measurement, 19, Sackett, P. R., & Larson, J. R. Jr. (1990). Research strategies and tactics in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology, (2 nd ed.), Vol. 1, p Palo Alto, CA: Consulting Psychologists Press. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores (Psychometric Monograph No. 18). Iowa City, IA: Psychometric Society. Schwarz, N. (1995). Social cognition: Information accessibility and use in social judgment. In E. E. Smith & D. N. Oshenson (Eds.), Thinking: An Invitation to Cognitive Science, Second Edition: Cambridge Mass: MIT Press. Schwarz, N., & Bless, H. (1992). Assimilation and contrast effects in attitude measurement: An inclusion/exclusion model. Advances in Consumer Research, 19, Stanton, J. M., Sinar, E. F., Balzer, W. K., & Smith, P. C. (2002). Issues and strategies for reducing the length of self-report scales. Personnel Psychology, 55, Thissen, D. (1991). MULTILOG users guide: Multiple categorical item analysis and test scoring using item response theory (computer program). Chicago,IL: Scientific Software International. Tourangeau, R. (1999) Context effects on answers to attitude questions. In M. G. Sirken, D. J. Herrmann, S. Schechter, N. Schwarz, J. M Tanur, R. Tourangeau. Cognition and Survey Research, pp New York: John Wiley and Sons. Tourangeau, R., & Rasinski, K. A. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103, Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response.: Cambridge University Press. Tourangeau, R., Singer, E., & Presser, S. (2003). Context effects in attitude surveys: Effects on remote items and impact on predictive validity. Sociological Methods & Research, 31, Wells, C. S., Subkoviak, M. J., & Serlin, R. C. (2002). The effect of item parameter drift on

7 Context Effects & IRT 7 examinee ability estimates. Applied Psychological Measurement, 26, Author Contact Info: Drew Rivers Department of Psychology North Carolina State University Campus Box 7650 Raleigh, NC Phone: Fax: dcrivers@ncsu.edu Adam W. Meade Department of Psychology North Carolina State University Campus Box 7650 Raleigh, NC Phone: Fax: awmeade@ncsu.edu

8 Context Effects & IRT 8 Table 1 Items Forming The Goals And Direction Scale 1 I agree with the directions and plans for this company. 2 I understand the directions and plans for this company. 3 [This Company] is effective at integrating all of its various work locations into one unified company. 4 This company is making the changes necessary to compete effectively. 5 I believe this company appropriately balances the needs of its customers, employees and investors. 6 I am confident of the ability of senior management (e.g., the top 150 leaders within [The Company] to make decisions necessary to ensure the future success of [The Company] 7 High ethical standards are practiced by this company's management. 8 This company encourages employee involvement in the communities in which it operates. 9 Communications I receive from management help me understand [The Company s] business strategies and values. 10 (Target How satisfied are you with the information you receive from management on what's going on at this company? Item) Note: The first nine items utilized a 5-point, strongly disagree to strongly agree scale. Item ten utilized a 5-point very dissatisfied to very satisfied scale.

9 Context Effects & IRT 9 Table 2 Response Frequencies (Percentages), Means, and Standard Deviations for 2003 Sample (N=10440) and 2004 Sample (N=6651) Groups Category Percentages Item Mean sd Sample Sample Sample Sample Sample Sample Sample Sample Sample Sample Sample Sample Sample Sample Sample Sample Sample Sample Sample Sample

10 Context Effects & IRT 10 Table 3 CDIF and NCDIF Results Item CDIF NCDIF DTF Note: CDIF=Compensatory Differential Item Functioning; NCDIF=Non-compensatory Differential Item Functioning

11 Context Effects & IRT 11 Table 4 Equated Parameter Estimates for Item 10 Parameter estimates Time period a b1 b2 b3 b Sample Sample

12 Context Effects & IRT 12 Table 5 Target Item Context 2003 Survey Form Context Item 1 Context Item 2 As an employee of [Company Name], you receive a variety of information in a variety of ways. In general, do you feel you receive: Too much information About the right amount of information Too little information Where you usually hear general news about [Company Name] first? Click on the arrow below to select your first choice. Printed company publications Location or departmental communications My immediate supervisor/manager Employee Portal/Intranet Word of mouth/employee "grapevine" External Resources (e.g. Internet, newspaper, TV) Target Item How satisfied are you with information you receive from management on what s going on in the company? Very Satisfied Satisfied Neither Satisfied nor Dissatisfied Dissatisfied Very Dissatisfied. NO RESPONSE

13 Context Effects & IRT 13 Table 6 Target Item Context 2004 Survey Form Context Item 1 Context Item 2 Target Item As an employee of [Company Name], you receive a variety of information in a variety of ways. In general, do you feel you receive: Too much information About the right amount of information Too little information The company s communications to employees are straightforward and credible. Strongly Agree Agree Neither Agree nor Disagree Disagree Strongly Disagree No Response How satisfied are you with information you receive from management on what s going on in the company? To A Very Great Extent To a Great Extent To Some Extent To a Little Extent To Very Little or No Extent No Response

14 Context Effects & IRT 14 Figure 1. Trend in favorable ratings, Percent Favorable Year

15 Context Effects & IRT 15 Figure 2. Scree Plot for 10-item Scale 6 5 Eigenvalue Factor

16 Context Effects & IRT 16 Figure 3. Boundary Response Functions for Item P(theta) Theta

17 Context Effects & IRT 17 Figure 4. Category Response Curves for Item P(theta) Theta 2003 P(theta) Theta