Psychometric properties of the Irish Management Standards Indicator Tool

Transcription

1 Occupational Medicine 2016;66: doi: /occmed/kqw163 Psychometric properties of the Irish Management Standards Indicator Tool S. Boyd 1, R. Kerr 2 and P. Murray 3 1 Council for the Homeless Northern Ireland (CHNI), 4th Floor, Andras House, 60 Great Victoria Street, Belfast BT2 7BB, UK, 2 Department of Management and Leadership, University of Ulster, Ulster Business School, Jordanstown, County Antrim BT37 0QB, Northern Ireland, 3 Health and Safety Authority, Dublin D01 K0Y8, Ireland. Correspondence to: S. Boyd, CHNI, 4th Floor, Andras House, 60 Great Victoria Street, Belfast BT2 7BB, UK. Tel: +44 (0) ; fax: +44 (0) ; admin@chni.org.uk Background Work Positive is Ireland s national policy initiative to control work-related stress. Since the introduction of the UK Health and Safety Executive s Management Standards (MS) in 2004, a number of studies have been undertaken to assess the potential adaptation of the MS framework within Ireland. Aims Methods Results To investigate the dimensionality, reliability and validity of the Irish version of the MS Indicator Tool (ROI-MSIT). Between February 2011 and June 2014, we collected data from a wide range of public and private sector organizations that used the ROI-MSIT. In addition to the ROI-MSIT, respondents completed the WHO-Five Well-being Index (WHO-5). Exploratory factor analysis (EFA) was used to determine whether the ROI-MSIT maintained the structure of the UK instrument. The internal consistency of the ROI-MSIT was also assessed to determine its reliability, while its criterion-related validity was explored through correlation analysis with the WHO-5. Data were collected from 7377 participants. The factor structure of the ROI-MSIT consisted of six factors; the Demands, Control, Peer Support, Relationships and Role factors were equivalent to the original UK factors. Like the Italian version, a principal factor emerged that combined the Manager Support and Change domains. Cronbach s alpha scores ranged from 0.75 to Finally, the ROI- MSIT s subscales and WHO-5 were positively correlated (r = , P < 0.001). Conclusions The ROI-MSIT is reliable and valid, with a factor structure similar to the original UK instrument and the Italian MSIT. Further psychometric evaluation of the ROI-MSIT is recommended. Key words Introduction Factor analysis; indicator tool; management standards; psychometrics; psychosocial hazards; risk assessment. Work-related stress is a widespread problem in the Republic of Ireland (ROI), ranked second only to musculoskeletal disorders amongst the main sources of occupational illness [1]. In response to the growing concern surrounding this issue, the Irish Health and Safety Authority (HSA) launched a national policy initiative Work Positive in 2002 to help control levels of workrelated stress in the ROI. Since the introduction of the UK Health and Safety Executive s Management Standards in 2004, the Work Positive initiative has been revisited and revised on a number of occasions; consequently, the latter initiative now incorporates the full methodology of the former. For example, between 2004 and 2005, the HSA, the Health Education Board for Scotland and the UK Health and Safety Executive (HSE) jointly commissioned work to update Work Positive to include the Management Standards and make other amendments considering the findings of previous evaluations of the initiative. Since then, the HSA has continued to commission studies to evaluate the application and adaptation of the Management Standards methodology in Ireland. According to the UK HSE, the Management Standards describe a set of conditions that reflect high levels of employee well-being and organizational performance [2]. The specific working conditions that the Management Standards address are organized according to six psychosocial domains (Demands, Control, Support, Relationships, Role and Change). As an official accompaniment to the initiative, organizations can use the Management Standards Indicator Tool, the MSIT The Author Published by Oxford University Press on behalf of the Society of Occupational Medicine. All rights reserved. For Permissions, please journals.permissions@oup.com

2 720 OCCUPATIONAL MEDICINE (or UK-MSIT), to assess their performance against the Management Standards. Since 2004, a number of studies have investigated the psychometric properties of the MSIT [3 5], the results of which indicate that the instrument demonstrates sufficient reliability and validity for use in UK organizations. In more recent years, interest in the Management Standards initiative has spread beyond the UK and a number of international versions of the MSIT have been created as a result [6]. In Italy, for example, the findings from the evaluation of the Italian MSIT have been published, beginning in In summary, the results of several validation studies demonstrate that the Italian version of the MSIT behaves similarly to the UK-MSIT, although the domains of Manager Support and Change may not be as conceptually distinct for Italian workers. Magnavita [7], for example, has reported that an EFA of the Italian questionnaire revealed a merged factor consisting of all the Manager Support and Change items combined plus one Control item. The remaining five factors corresponded to the original UK domains (Demands, Control, Peer Support, Relationships and Role). Cronbach s alphas for the six subscales ranged from 0.75 to In a separate study by Rondinone et al. [8] using confirmatory factor analysis (CFA), the original seven-factor measurement model of the UK-MSIT offered a better fit to the Italian data than the six-factor structure uncovered by Magnavita [7]. Finally, Guidi et al. [9] found that the Italian MSIT s subscales were negatively correlated with the 12-item General Health Questionnaire and positively correlated with the Work Ability Index. International interest in the Management Standards methodology is growing and, consequently, scholars are now requesting additional research on the MSIT s reliability and validity, particularly in countries outside of the UK [10,11]. Considering both the research needs of the Irish HSA and the international interest in the Indicator Tool, the aim of the current study was to investigate the psychometric properties of the newly developed Irish version (English language) of the MSIT (ROI-MSIT), known in Ireland as the Work Positive Profile. This paper reports the results and findings of the study, which explored the factor structure, internal consistency and criterion-related validity of the questionnaire. Methods The 35-item ROI-MSIT was used by each organization to measure employees perceptions of workplace stressors. Each item of the ROI-MSIT is scored on a 5-point Likert scale requiring respondents to indicate either the frequency of an event or the degree of their agreement with a statement. A score of 5 is the highest (most positive) rating a respondent can give for each item. Items pertaining to the domains of Demands and Relationships are scored negatively. Taken at face value, the ROI-MSIT is almost identical to the original UK-MSIT, although the wording of several of its items was changed slightly by the Irish HSA prior to commencing the study. Though the amendments may appear very superficial, it cannot be taken for granted that the ROI-MSIT s items measure the exact same constructs as the original UK Indicator Tool, since even minor amendments can significantly alter a question s meaning for respondents [12]. The WHO-Five Well-being Index (WHO-5) was also included in the study to assess the ROI-MSIT s criterionrelated validity. An abbreviated form of the WHO Well- Being Scale, the WHO-5, measures the degree of positive well-being present in the last 2 weeks across five items [13,14]. Each item is scored on a 6-point Likert scale ranging from 0 (not present) to 5 (constantly present). The total raw score ranges from 0 to 25; scores below 13 indicate possible depression and a reduced quality of life. The WHO-5 has proven to be a valid and reliable instrument in a wide variety of settings and amongst diverse patient and population groups, including managers [15]. To investigate the dimensionality of the ROI-MSIT, EFA was applied to the data. Although the structure of the original UK-MSIT is known to comprise seven factors, an exploratory approach was deemed necessary considering that (i) the MSIT had not been psychometrically assessed using Irish data before, and (ii) the HSA had made changes to the wording of the original UK instrument. EFA was therefore performed to examine the structure without the constraint of an a priori model. Before conducting the EFA, however, the Kaiser Meyer Olkin test and Bartlett test of sphericity were applied to the data set to assess its suitability for factor analysis; the results indicated that the EFA could proceed [16]. To that end, the method reported by Cousins et al. [17] in their study of the UK-MSIT was used, specifically principal component analysis (PCA) with promax rotation. The use of promax rotation was also justified on the grounds that an oblique method of rotation is best suited to instruments where there is a known overlap between factors [16,18], as is the case with the UK-MSIT. It was hypothesized that the EFA s pattern matrix would replicate the original seven-factor solution of the UK-MSIT. To determine the number of factors retained, the Kaiser Guttman test was applied so that only factors with an eigenvalue of at least 1.00 were kept in the analysis. The results of Cattell s scree test were also taken into consideration, since it is recommended that more than one test is used for the purposes of factor retention [18]. Cronbach s alphas were calculated to measure the ROI- MSIT s internal consistency; it was hypothesized that each subscale would achieve a score of at least The criterion-related validity of the ROI-MSIT was then investigated through correlation analysis with the WHO-5. Accordingly, Pearson s correlations (two-tailed) were calculated to examine associations between the ROI-MSIT s subscales

3 S. BOYD ET AL.: PSYCHOMETRIC PROPERTIES OF THE IRISH MSIT 721 and the well-being measure; it was hypothesized that better (higher) scores on the ROI-MSIT s subscales would be associated with better (higher) scores on the WHO-5. All statistical analyses were performed in SPSS v. 21. Ethical approval for the study was granted from the Ethics Committee of the University of Ulster. Results A total of 7377 employees completed the ROI-MSIT as part of the HSA s latest project evaluating the application of the Management Standards in Ireland (February 2011 to June 2014). The project used volunteer sampling to obtain a diverse convenience sample of 35 organizations; a wide range of private and public sector organizations were included in the sample. The average sample size across the organizations was 210 respondents. Pairwise deletion was adopted for all statistical analyses. In respect of the pooled data set analysed in the study, 58% of the respondents were men, while women made up 42% of the sample. The majority of respondents (64% of the sample) were aged between 36 and 55, and 51% had been working in their organization for over 10 years. Approximately a third of respondents (34%) had been working in their organization for 3 10 years. The Kaiser Meyer Olkin test and Bartlett test of sphericity resulted in scores of 0.95 and (df = 595, P < 0.001), respectively, indicating that the EFA could proceed. PCA was used to extract the factors. On the basis of the Kaiser Guttman rule (eigenvalues > 1.00) and Cattell s scree test, six factors were retained in the final solution, which together accounted for 60% of the ROI-MSIT s total variance. The retained factors were then rotated using promax rotation. The results of the EFA are presented in Table 1, which shows the eigenvalues and item loadings for the factors extracted. All item loadings were greater than 0.40, exceeding the recommended minimum loading score of 0.32 [16]. As can be seen in Table 1, the principal factor uncovered by the EFA was a merged factor consisting of all the Manager Support and Change items (this merger was simply labelled Manager Support-Change ). The loading for each of these items was strong ( 0.65), suggesting that the factor is indeed a distinct domain. Apart from the amalgamation of the Manager Support and Change components, the ROI-MSIT s factor structure was almost identical to that of the original UK-MSIT. However, it is worth noting that two items cross-loaded onto another factor; the first of these crossloading items was a Control question (Q. 10): I have enough control over the pace of my work, which had a loading of 0.40 on the Control factor and a cross-loading of 0.44 on the Demands factor. The second was a Peer Support question (Q. 27): Colleagues generally treat me respectfully at work, which had a loading of 0.48 on the Peer Support factor and a cross-loading of 0.42 on the Relationships domain. It was decided that these cross-loading items would be retained under their original Management Standards domains in subsequent analyses. Following the EFA, descriptive statistics (means and standard deviations) and Cronbach s alpha coefficients were calculated for the data of the pooled sample (n = 7377). In keeping with the factor structure uncovered by the EFA, subsequent analyses of the ROI-MSIT treated it as a measure consisting of six subscales. The two cross-loading items were retained within their original subscales, as discussed previously. Alphas for the ROI-MSIT s six subscales and the WHO-5 are presented in Table 2; the corresponding scores for the Italian and UK instruments are also included in the table. The alpha coefficients ranged from 0.75 to 0.91 and were comparable to those reported for the British and the Italian versions [4,7,17]. Finally, the alpha coefficient for the WHO-5 was 0.90, a score consistent with previous studies of the scale [19]. Table 3 presents the two-tailed correlation coefficients for the six subscales and the WHO-5. The correlation analysis revealed highly significant, moderate strength correlations in a positive direction between each subscale and the WHO-5 (r = , P < 0.001). Finally, the results displayed in Table 3 also show that there was a significant positive correlation between each subscale and the other individual subscales (r = , P < 0.001). Discussion This preliminary study demonstrated that the Irish Indicator Tool (ROI-MSIT) has retained most of the structural characteristics of the original UK questionnaire. The Cronbach s alpha scores show that the ROI- MSIT is highly internally consistent and of comparable reliability to the UK and Italian Indicator Tools [4,7,17]. The results of the study also demonstrate the criterionrelated validity of the ROI-MSIT, as significant positive correlations were revealed between each of the subscales and the WHO-5, indicating that better (i.e. higher) scores across the Management Standards domains are associated with better mental health. Although very few studies have examined associations between the MSIT and measures of positive mental health, these correlations were of the expected significance, direction and magnitude considering the correlations reported between the UK-MSIT s subscales and another measure of positive mental health, the Warwick-Edinburgh Mental Well-being Scale [20]. However, the results of the EFA revealed significant differences between the factor structure of the ROI- MSIT and the original UK instrument. Although the Irish instrument retained most of the principal characteristics of the original MSIT in this analysis, the Manager Support and Change components were merged together instead of being separate domains. Taken at face value,

4 722 OCCUPATIONAL MEDICINE Table 1. Promax factor analysis of the ROI-MSIT Scale and item Loading Manager Support-Change (λ 10.84) a Q. 35 My line manager encourages me at work 0.88 Q. 8 I am given supportive feedback on the work I do by my line manager 0.82 Q. 29 I feel I can talk to my line manager about something that has upset or annoyed me about work 0.76 Q. 23 I can rely on my line manager to help me out with a work-related problem if I ask 0.73 Q. 33 I feel I would be supported by management if I had emotionally demanding work 0.73 Q. 32 When changes are made at work, it is explained how they will work out in practice 0.72 Q. 28 Staff are always consulted about change at work 0.71 Q. 26 I have sufficient opportunities to question management about change at work 0.66 Demands (λ 3.34) a Q. 20 I have to work too fast 0.80 Q. 22 I have unrealistic time pressures 0.79 Q. 9 I have to work very intensively 0.77 Q. 12 I have to neglect some tasks because I have too much work to do 0.75 Q. 18 I am pressured to work long hours 0.73 Q. 16 I feel I cannot take enough break time 0.67 Q. 6 I have unachievable deadlines 0.65 Q. 3 Different groups at work demand things from me that I find hard to combine 0.62 Role (λ 2.12) a Q. 11 I am clear in what my duties and responsibilities are 0.85 Q. 1 I am clear in what is expected of me at work 0.81 Q. 4 I know how to go about getting my job done 0.80 Q. 13 I am clear about the goals and objectives for my department/work group 0.73 Q. 17 I understand how my work fits into the overall aim of the organization 0.67 Control (λ 1.97) a Q. 15 I have a choice in deciding how I do my work 0.80 Q. 19 I have a choice in deciding what I do at work 0.76 Q. 2 I can decide when to take a break 0.76 Q. 25 I have some say over the way I work 0.65 Q. 30 My working time can be flexible 0.57 Q. 10 I have enough control over the pace of my work 0.40 (0.44) b Relationships (λ 1.51) a Q. 21 I am subject to bullying at work 0.84 Q. 5 I am subject to personal harassment in the form of unkind words or behaviour at work 0.80 Q. 14 There is friction or anger between colleagues 0.65 Q. 34 Relationships at work are strained 0.59 Peer Support (λ 1.15) a Q. 7 If work gets difficult, my colleagues will help me if I ask 0.82 Q. 24 I get the help and support I need from my colleagues if I ask 0.79 Q. 31 I feel I can talk to my colleagues to solve work-related issues 0.67 Q. 27 Colleagues generally treat me respectfully at work 0.48 (0.42) b a Eigenvalue score. b Figures in brackets represent cross-loadings onto another factor. it is not difficult to appreciate the conceptual similarities between these two domains, as organizational change is typically decided at a managerial level and communicated to workers by line management. On the other hand, the conceptual similarities between these subscales do not offer a satisfactory answer as to why the two domains have merged in the ROI-MSIT and not the UK instrument. Interestingly, an EFA (varimax rotation) of the Italian MSIT by Magnavita [7] revealed an almost identical six-factor structure; indeed, the Manager Support and Change domains were combined as a merged factor, which also included one Control item (Q. 30). In a subsequent CFA of the Italian MSIT by Rondinone et al. [8], the CFA revealed that the original seven-factor measurement model of the UK Indicator Tool provided the best fit to the data, outperforming the six-factor model reported by Magnavita [7]. The logical implication of this finding is that the emergence of a six-factor structure through EFA is not, by itself, sufficient evidence of a divergence in dimensionality from the original UK-MSIT. Furthermore, it should be acknowledged that the factor structure presented in this paper lends credence to the claims of Magnavita [7], who has argued that the sixfactor pattern he uncovered for the Italian Indicator Tool is likely a result of cultural and social differences between Britain and Italy. Considering that a near-identical factor

5 S. BOYD ET AL.: PSYCHOMETRIC PROPERTIES OF THE IRISH MSIT 723 Table 2. Cronbach s alpha coefficients and descriptive statistics for the ROI-MSIT and WHO-5 Scale Subscale No. of items Mean SD Alpha Alpha (IT-MSIT) Alpha (UK-MSIT) ROI-MSIT MS-Change a b Demands Role Control Relationships Peer Support WHO The corresponding alpha scores for the Italian MSIT (IT-MSIT) and UK MSIT (UK-MSIT) are listed on the right-hand side for comparative purposes. a Manager Support-Change (amalgamated factor). b No corresponding score for the UK-MSIT. Table 3. Pearson s product moment correlation coefficients (two-tailed) between the six ROI-MSIT subscales and the WHO-5 MS-Change a Demands Role Control Relationships Peer Support WHO-5 MS-Change 1.00 Demands 0.44*** 1.00 Role 0.57*** 0.44*** 1.00 Control 0.64*** 0.44*** 0.46*** 1.00 Relationships 0.66*** 0.53*** 0.47*** 0.53*** 1.00 Peer Support 0.74*** 0.44*** 0.48*** 0.58*** 0.68*** 1.00 WHO *** 0.43*** 0.42*** 0.49*** 0.52*** 0.58*** 1.00 Values in bold represent a perfect correlation between each subscale with itself. a Manager Support-Change (amalgamated factor). ***P < solution was produced for the ROI-MSIT, it seems probable that cultural differences are indeed responsible for the six-factor pattern uncovered through EFA in both instances. The replication of this six-factor structure in the current study may, therefore, be a result of minor cultural differences amongst the Irish sample and not a lack of validity with the UK-MSIT. CFA is now required to test the comparative fit of the six- and seven-factor measurement models. If the sixfactor model of the ROI-MSIT provides the best fit in an upcoming CFA study, the instrument will have to be treated as a six-subscale measure. Although the study provides evidence of the reliability and validity of the ROI-MSIT using a large and diverse sample of Irish workers (n = 7377), several methodological weaknesses should be borne in mind when interpreting the results. In particular, the study s crosssectional survey design did not enable a thorough investigation of the relationships between the ROI-MSIT and WHO-5. Although the associations revealed in the correlation analysis were significant and in the expected direction, causality cannot be assigned to any of them in the absence of a longitudinal study. A prospective study should therefore be conducted in future to identify any causal links among these associations and the nature of their direction, including reciprocal effects. The use of non-probabilistic convenience sampling limits the generalizability of the results and findings; on the other hand, this threat to external validity is not a particular concern, considering that the study s aim was to assess the internal psychometric characteristics of the ROI-MSIT and not the external characteristics of Irish organizations or the Irish working population in general. In conclusion, the results of this preliminary investigation of the ROI-MSIT s psychometric properties indicate that the questionnaire is of sufficient reliability and validity for use in Ireland by both practitioners and researchers. Key points The exploratory factor analysis of this study revealed a six-factor pattern for the Irish Management Standards Indicator Tool that is almost identical to that reported of the Italian Management Standards Indicator Tool. The Irish Management Standards Indicator Tool is highly internal consistent and demonstrated its criterion-related validity with the WHO-Five Wellbeing Index. The Irish Management Standards Indicator Tool is of sufficient psychometric quality for use in the Republic of Ireland (although further analysis of its measurement structure through confirmatory factor analysis is strongly recommended).

6 724 OCCUPATIONAL MEDICINE Funding Department for Employment and Learning Northern Ireland (PhD Studentship award to SB); Irish Health and Safety Authority. Conflicts of interest None declared. References 1. Bevan S, Quadrello T, McGee R, Mahdon M, Vavrovsky A, Barham L. Fit for Work? Musculoskeletal Disorders in the European Workforce. London, UK: The Work Foundation, Health and Safety Executive [HSE]. Managing the Causes of Work-Related Stress: A Step-by-Step Approach Using the Management Standards. Norwich, UK: HSE Books, Main C, Glozier N, Wright I. Validity of the HSE stress tool: an investigation within four organizations by the Corporate Health and Performance Group. Occup Med (Lond) 2005;55: Edwards J, Webster S. Psychosocial risk assessment: measurement invariance of the UK Health and Safety Executive s Management Standards Indicator Tool across public and private sector organizations. Work Stress 2012;26: Kerr R, McHugh M, McCrory M. HSE management standards and stress-related work outcomes. Occup Med (Lond) 2009;59: Mackay C, Palferman D. Policy level interventions for organizational health: development and evolution of the UK Management Standards. In: Bauer G, Jenny G, eds. Salutogenic Organizations and Change: The Concepts Behind Organizational Health Intervention Research. Dordrecht, The Netherlands: Springer, 2013; Magnavita N. Validation of the Italian version of the HSE Indicator Tool. Occup Med (Lond) 2012;62: Rondinone BM, Persechino B, Castaldi T et al. Workrelated stress risk assessment in Italy: the validation study of health safety and executive indicator tool. G Ital Med Lav Ergon 2012;34: Guidi S, Bagnara S, Fichera GP. The HSE indicator tool, psychological distress and work ability. Occup Med (Lond) 2012;62: Brookes K, Limbert C, Deacy C, O Reilly A, Scott S, Thirlaway K. Systematic review: work-related stress and the HSE management standards. Occup Med (Lond) 2013;63: Toderi S, Balducci C, Edwards J, Sarchielli G, Broccoli M, Mancini G. Psychometric properties of the UK and Italian versions of the HSE stress indicator tool. Eur J Psychol Assess 2013;29: Behling O, Law K. Translating Questionnaires and Other Research Instruments: Problems and Solutions. London, UK: Sage, Awata S, Bech P, Yoshida S et al. Reliability and validity of the Japanese version of the World Health Organization- Five Well-Being Index in the context of detecting depression in diabetic patients. Psychiatry Clin Neurosci 2007;61: Bech P. Measuring the dimensions of psychological general well-being by the WHO-5. QoL Newsletter 2004;32: Bech P, Andersen MB, Bech-Andersen G, Tønnesen S, Agnarsdottir E, Borg V. Work-related stressors, depression and quality of life in Danish managers. Eur Psychiatry 2005;20(Suppl. 3):S318 S Tabachnick B, Fidell L. Using Multivariate Statistics. Edinburgh, UK: Pearson, Cousins R, Mackay C, Clarke S, Kelly C, Kelly P, McCaig R. Management standards work-related stress in the UK: practical development. Work Stress 2004;18: Hair J, Black W, Babin B, Anderson R, Tatham R. Multivariate Data Analysis. Upper Saddle River, NJ: Pearson Education, Zierau F, Bille A, Rutz W, Bech P. The Gotland Male Depression Scale: a validity study in patients with alcohol use disorder. Nord J Psychiatry 2002;56: Bartram DJ, Yadegarfar G, Baldwin DS. Psychosocial working conditions and work-related stressors among UK veterinary surgeons. Occup Med (Lond) 2009;59: