Evaluation of the Pharmacy Safety Climate Questionnaire in European community pharmacies

Transcription

1 International Journal for Quality in Health Care 2012; Volume 24, Number 1: pp Advance Access Publication: 2 December 2011 Evaluation of the Pharmacy Safety Climate Questionnaire in European community pharmacies /intqhc/mzr070 DENHAM L. PHIPPS 1,8, JOLANDA DE BIE 2, HANNE HERBORG 3, MARA GUERREIRO 4, CHRISTIANE EICKHOFF 5, FERNANDO FERNANDEZ-LLIMOS 6, MARCEL L. BOUVY 2,7, CHARLOTTE ROSSING 3, UTA MUELLER 5 AND DARREN M. ASHCROFT 1 1 School of Pharmacy and Pharmaceutical Sciences, University of Manchester, UK, 2 SIR Institute for Pharmacy Practice and Policy, Leiden, The Netherlands, 3 Pharmakon, Hillerød, Denmark, 4 Research Institute for Medicines and Pharmaceutical Sciences, University of Lisbon (imed-ul); Instituto Superior de Ciências da Saúde Egas Moniz (ISCSEM), Portugal, 5 ABDA Federal Union of German Associations of Pharmacists, Berlin, Germany, 6 Department of Social Pharmacy, Faculty of Pharmacy, University of Lisbon, Lisbon, Portugal, and 7 Department of Pharmacoepidemiology and Clinical Pharmacology, Utrecht University, The Netherlands Address reprint requests to: Denham L. Phipps, Human Reliability Associates Ltd, UK. phippsdl@gmail.com Accepted for publication 3 November 2011 Abstract Objective. To evaluate the internal reliability, factor structure and construct validity of the Pharmacy Safety Climate Questionnaire (PSCQ) when applied to a pan-european sample of community pharmacies. Design. A cross-sectional survey design was used. Setting. Community pharmacies in Denmark, Germany, the Netherlands, Portugal and Great Britain. Participants. A total of 4105 members of the community pharmacy workforce, all drawn from one of the five participating countries. Main outcome measures. Each participant completed a copy of the Pharmacy Safety Climate Questionnaire in his or her respective language and rated the perceived safety of the pharmacy in which he or she worked. Results. Exploratory and confirmatory factor analyses of the data identified four factors that accounted for item responses, with 24 of the original 34 items loading onto them. They were labelled organizational learning, blame culture, working conditions and safety focus. These factors were found to have an acceptable level of reliability (with Cronbach s alpha values ranging from 0.70 to 0.92) and to predict the rating of pharmacy safety. Conclusions. This study provided information on the PSCQ s psychometric properties when used in community pharmacies in different European countries. A modified version of the original PSCQ (known as PSCQ-4) is presented, and further work is proposed to demonstrate its application to safety improvements in pharmacies. Keywords: pharmacy, safety climate, safety culture, psychometric, questionnaire Introduction Knowledge transfer from high-reliability organizations to the healthcare sector has led to the latter taking an increasing interest in the notion of safety culture and safety climate [1]. Safety culture can be defined in simple terms as the set of values and beliefs that a given organization holds with regard to safety; these beliefs manifest themselves as a safety climate, which is a corresponding set of attitudes and behavioural 8 Present address: Human Reliability Associates Ltd, Dalton, Lancashire, UK. norms as perceived by people who work in the organization [2, 3]. The importance of safety culture and climate is that it provides a basis for understanding why staff within a given work setting behave in a safe or unsafe manner. The data from these assessments can be put to a variety of uses, for example, diagnosis and awareness raising, evaluation, benchmarking and audit [4]. There are a number of ways in which this can be assessed; typically, cross-sectional surveys of climate (as the more tangible of the two) are employed [5]. International Journal for Quality in Health Care vol. 24 no. 1 # The Author Published by Oxford University Press in association with the International Society for Quality in Health Care; all rights reserved 16

2 Evaluation of the PSCQ Safety measurement methods Table 1 Sample composition and demographic data However, the divergence of views about the specific nature of safety climate [3, 6] is reflected in the assortment of assessment instruments in healthcare, all varying in their scope, ease of use and measurement properties [7]. The Manchester Patient Safety Assessment Framework (MaPSAF) [8] provides an assessment approach that has a sound theoretical rationale as well as practical utility. This framework, which was developed through literature review and qualitative data from healthcare practitioners, forms the basis of a safety culture assessment that has been adapted for a number of healthcare domains, one of which is community pharmacy [9]. The Pharmacy Safety Climate Questionnaire (PSCQ) [5] comprises 34 items, which correspond to seven dimensions of safety culture: investigating and learning; staffing and management; incident causation and reporting; team working; communication; commitment to patient safety; education and training. These dimensions were derived from a pilot study of the questionnaire [5]. However, the data for this study were drawn from a limited sample, namely attendees at a postgraduate training event in Great Britain. While the dimensions identified are applicable to this sample, it is not clear whether they can be generalized across a broader range of community pharmacies. The aim of the current study was to examine the structure and psychometric properties (reliability and validity) of the PSCQ items when applied to a general sample of pharmacy staff across five European countries. Method Study design and sampling This study used a cross-sectional survey design. The study population was community pharmacy staff in five European DNK DEU NLD PRT GBR Gender Male N (%) 40 (20.3) 540 (47.2) 137 (40.5) 133 (23.3) 779 (42.0) Female N (%) 152 (77.2) 517 (45.2) 189 (55.9) 411 (72.1) 1073 (57.8) No response N (%) 5 (2.5) 87 (7.6) 12 (3.6) 26 (4.6) 4 (0.2) Role Proprietor/manager N (%) 72 (36.5) 875 (76.5) (22.5) 532 (28.7) Pharmacist N (%) 100 (50.8) 138 (12.1) (72.5) 1211 (65.2) Technician/assistant N (%) 22 (11.2) 51 (4.5) Other (e.g. superintendent) N (%) 2 4 (0.3) (5.8) No response N (%) 3 (1.5) 76 (6.6) 338 (100) 29 (5.1) 5 (0.3) Age Mean (SD) 48.9 (10.2) 47.9 (10.2) 42.4 (9.8) 32.2 (8.9) 47.4 (11.3) Years of professional experience Mean (SD) 18.8 (11.7) 21.4 (10.8) 14.8 (8.7) 7.0 (7.4) 22.2 (11.8) Note: DNK, Denmark (Convenience sample of pharmacy staff with responsibility for patient safety activities. Online survey, N ¼ 197). DEU, Germany (Convenience sample of community pharmacies with contact details. Online survey, N ¼ 1144). NLD, The Netherlands (Random sample of members of the Royal Dutch Pharmacists Association consisting of proprietors, managers and pharmacists. Online survey, N ¼ 338, response rate 39%). PRT, Portugal (Convenience sample of proprietors, managers and pharmacists. Online survey, N ¼ 570). GBR, Great Britain (Convenience sample of attendees on a pharmacy risk management training course. Postal survey, N ¼ 998. Random sample of registered community pharmacists. Postal survey, N ¼ 858, response rate 42.9%). countries: Denmark, Germany, the Netherlands, Portugal and Great Britain. The sampling frame for each country is provided in the footnote to Table 1. Materials The PSCQ is a self-rated questionnaire that comprises a series of statements about the pharmacy s collective attitudes and behaviours regarding patient safety. The respondent rates his or her level of agreement with each statement on a fivepoint Likert scale that ranges from strongly disagree to strongly agree. This study used the original 34-item version of PSCQ developed by Ashcroft and Parker [5]. For administration to the sampling frames outside of Britain, the PSCQ was translated into the respective languages. In order to do so, researchers with knowledge of English and the target language translated the PSCQ into the latter. Other researchers then independently translated the alternative language versions back into English, and these back-translations compared against the original English version. Any discrepancies that had occurred during translation (e.g. misinterpretation of colloquialisms) were resolved by consultation with the translators. Procedure Participants within each sampling frame were contacted by the researchers and invited to complete the PSCQ. On completion, the PSCQ was returned directly to the researchers based at the local study site. While Great Britain used a postal version of the PSCQ, the other countries used online administration via the Internet. Participants in Denmark had the option of either paper or online administration. 17

3 Phipps et al. Results Preliminary analysis Data from completed questionnaires were entered into version 15 of the SPSS program for analysis. Data entry took place at each study site, with the files from each site then being combined into a master file at the British study site for analysis. Demographic data for the sample are shown in Table 1. Of 155,990 data points, 2927 (1.8%) were missing; this is a sufficiently small proportion to allow multivariate analyses to be carried out [10]. The seven-factor structure suggested by Ashcroft and Parker [5] was applied to the entire dataset using confirmatory factor analysis. This analysis was performed using version 16 of the AMOS program. In order to preserve cases in the sample, missing data were substituted using expectation maximization [11]. The model was fit to covariance matrices (which are available on request from the first author) and the maximum likelihood procedure was used to estimate model parameters. Model fit was assessed using the chi-square test, Tucker Lewis index (TLI), comparative fit index (CFI) and root mean-squared error of approximation (RMSEA). Hu and Bentler [12] suggest that TLI and CFI values close to or exceeding 0.95, and RMSEA values less that 0.06, are indicative of a good fit. Using these criteria, the fit between the model and the data was questionable [x 2 (511) ¼ , P, 0.001; TLI ¼ 0.83; CFI ¼ 0.85; RMSEA ¼ 0.07], suggesting that the seven-factor structure may not generalize very well beyond the sample from which it was originally derived. What factors are present in the questionnaire responses? In order to identify a new factor structure, the dataset was divided into two groups, with each participant assigned to one of the groups at random. The data from Group 1 (N ¼ 2055) were subjected to an exploratory factor analysis using SPSS. The extraction method was alpha factoring with listwise deletion of missing data. Three factors were obtained with eigenvalues greater than one; an additional fourth factor was extracted with an eigenvalue of These four factors accounted for 58.3% of the variance in the item responses. Bartlett s test of sphericity and the Kaiser2Meyer2Olkin measure of sampling adequacy were within acceptable limits [Bartlett x 2 (276) ¼ , P, 0.001; Kaiser2 Meyer2Olkin ¼ 0.969], indicating that the correlation matrix was suitable for analysis. Oblique rotation of the factors was performed using the oblimin and promax routines. The latter produced the item loadings shown in Table 2. These loadings suggest an interpretation of the factors as follows. The first factor contains items that reflect organizational learning; that is, the pharmacy s willingness to proactively develop and maintain a safe working environment (Cronbach s alpha reliability ¼ 0.92). The second factor alludes to the pharmacy s propensity to allocate blame to individuals when a problem or incident occurs (alpha ¼ 0.85). The third factor addresses the presence of working conditions that are conducive to safe working, for example, having sufficient personal and material resources to manage the workload [13, 14] (alpha ¼ 0.78). The fourth factor indicates the level of safety focus; in other words, the priority given to safety in the pharmacy s day-to-day work (alpha ¼ 0.69). Table 2 also indicates strong correlations between the four factors, especially between factor 1 and factors 2 and 4. This is consistent with the conceptual relationship between them. One would expect staff sensitivity about being blamed for incidents to be a barrier to organizational learning; similarly, any attempt to implement safety improvements would be undermined by a work environment that does not support safe working. Furthermore, an organization that is proactive about safety management is also likely to adopt a safety first approach in its everyday practices. During the factor analysis, 10 items, shown in Table 3, were discarded due to low loadings on all four factors, despite their initial communality values being comparable with those of the retained items. One item ( It s just luck that serious mistakes don t happen ) was retained despite being just short of the 0.30 loading value suggested as a minimum by Hair et al. [15]. As the sample is larger than that assumed by Hair et al. (N ¼ 2055 as opposed to 350), it was thought to be robust to a small departure from the required loading value. Hence, the factor solution comprises 24 of the original items. In order to test this four-factor structure, it was applied to the data from Group 2 (N ¼ 2050) using the same confirmatory factor analysis procedure as described earlier. The regression weights and covariances for this analysis are shown in Table 4. As this table shows, the paths between the factors and their indicator variables (the items) are statistically significant, providing support for the measurement model. Also, statistically significant covariance paths are observed between the factors; these broadly mirror the correlations observed in the exploratory factor analysis, although here, the strongest covariance is between blame culture and work conditions (the more the pressure is on staff, the greater the tendency to put blame on individuals for things that go wrong). The goodness-of-fit test was statistically significant [x 2 (246) ¼ , P, 0.001]. A significant goodness-of-fit result may be influenced by the large sample size; however, the other criteria, which are more resistant to differences in sample size, were close to their recommended cut-off values (TLI ¼ 0.94; CFI ¼ 0.95; RMSEA ¼ 0.05). When this model is fit to the entire sample, it generates an Akaike s information criterion value of This compares to a value of for the original seven-factor model, thus indicating that the new model is a comparatively better fit to the data. [16] (Akaike s information criterion was calculated using a correction for the ratio between number of parameters and sample size: AIC ¼ (x 2 2 d) þ [(2* K* (K þ 1))/(n 2 K 2 1)], where d is degrees of freedom, K is the number of parameters to be estimated and n is the sample size [17 18].) The new model will hereafter be referred to as PCSQ-4 to distinguish it from the original PSCQ. 18

4 Evaluation of the PSCQ Safety measurement methods Table 2 Item loadings from the exploratory factor analysis Item Factor Routinely discuss ways to prevent incidents 0.80 Assess risks and look for improvements 0.74 Staff are routinely informed about incidents 0.72 Culture of continuous improvement 0.69 Effectiveness of change is evaluated 0.66 Learn and share information about incidents 0.62 Shared understanding and vision about safety 0.61 Commitment to change following an incident 0.60 Staff feel free to speak up about safety matters 0.57 Investigations seen as learning opportunities 0.56 Investigations aim to learn from incidents 0.51 Manager considers staff suggestions 0.51 Staff have education and training in safety 0.46 There is a blame culture 0.80 Staff feel that mistakes are held against them 0.78 The person is reported rather than the problem 0.77 Investigations aim to blame individuals 0.73 There are enough staff to handle the workload 0.83 Staff try to do too much, too quickly Staff work longer hours than is sensible It s just luck that serious mistakes don t happen Training in safety has a low priority Staff are already trained to do their job Lip service is paid to safety until something happens Eigenvalue Percentage of variance Correlation with Factor Factor Factor Table 3 Items removed from the PSCQ-4, with communality values during the first iteration of the exploratory factor analysis Item Communality 3. Similar incidents tend to recur There are tensions between staff members in the pharmacy* Individuals are not actually committed to the team and only work together because they have to* Staff in the pharmacy are seen as the cause of safety incidents and the solution is retraining 0.57 and punitive action 21. Staff feel free to question the decisions or actions of those with more authority Patient safety is never sacrificed to get the work done The pharmacy welcomes any outside involvement in investigations Everyone in the pharmacy has confidence in the management Incidents and complaints are swept under the carpet if possible The pharmacy uses more locum/temporary staff than is sensible for patient care 0.38 Note: Asterisks indicate items that formed the Team Working dimension in the original model. Item numbers are from Ashcroft and Parker [5]. 19

5 Phipps et al. Table 4 Standardized regression weights and covariances for the confirmatory factor analysis Item What do the factor scores indicate about safety in community pharmacy across the study locations? In addition to the PSCQ items, respondents were asked to provide a single rating for the overall perceived safety of the pharmacy; the rating was on a five-point Likert scale from 1 ( Excellent ) to 5 ( Failing ). A regression analysis was carried out with this rating as the dependent variable and the PSCQ-4 scores as predictors. A statistically significant prediction was obtained from the scores [R ¼ 0.57, adj. R 2 ¼ 0.32, F(4,3283) ¼ , P, 0.001]. The Durbin2Watson statistic for this regression model was 1.96, and the residual values were normally distributed and homoscedastic. The regression coefficients are shown in Table 5; these indicate Factor Routinely discuss ways to prevent incidents 0.72 Assess risks and look for improvements 0.71 Staff are routinely informed about incidents 0.68 Culture of continuous improvement 0.78 Effectiveness of change is evaluated 0.60 Learn and share information about incidents 0.64 Shared understanding and vision about safety 0.74 Commitment to change following an incident 0.74 þ Staff feel free to speak up about safety matters 0.64 Investigations seen as learning opportunities 0.73 Investigations aim to learn from incidents 0.68 Manager considers staff suggestions 0.67 Staff have education and training in safety 0.52 There is a blame culture 0.80 Staff feel that mistakes are held against them 0.77 The person is reported rather than the problem 0.76 þ Investigations aim to blame individuals 0.77 There are enough staff to handle the workload 0.71 Staff try to do too much, too quickly 0.74 þ Staff work longer hours than is sensible 0.63 It s just luck that serious mistakes don t happen 0.71 Training in safety has a low priority 0.74 Staff are already trained to do their job 0.61 þ Lip service is paid to safety until something happens 0.66 Covariance with Factor 2 Estimate SE Factor 3 Estimate SE Factor 4 Estimate SE Note: Items loading negatively in the exploratory factor analysis were reverse scored, and the raw value for regression weights marked þ was fixed to 1. For all other values, P, that with the exception of blame culture (P ¼ 0.76), the PSCQ-4 scores all make a statistically significant contribution to the model. It should be noted from the table that blame culture has as strong a Pearson s correlation with the safety rating as do the other scores; that it does not contribute to the regression model is likely to be a result of multicollinearity between the predictors (cf. Tables 2 and 4). Due to a relatively high proportion of missing data (10% of responses) among the safety ratings, a second regression analysis was carried out on a dataset in which the missing responses were substituted with maximum likelihood estimates. The model generated from this dataset was similar to that generated from the original data (R ¼ 0.59, adj. R 2 ¼ 0.34, P, 0.001). 20

6 Evaluation of the PSCQ Safety measurement methods Table 5 Regression coefficients for the prediction of overall safety rating PSCQ score Beta t Pearson s r... Discussion Semi-partial r Organizational learning ** Blame culture Working conditions ** Safety focus * Note: *P, 0.01, **P, This study builds upon Ashcroft and Parker s [5] initial work to develop a safety climate assessment for pharmacies. While Ashcroft and Parker identified seven dimensions in their pilot data, this study found that a four-factor framework, consisting of 24 items, emerges when applied to a broader range of community pharmacy staff. The latter appears to provide a better account of the questionnaire data, and has an acceptable level of measurement reliability. Furthermore, a consistent relationship is found between scores on the factors and a rating of perceived overall safety. Therefore, it is recommended that the four-factor version of the PSCQ presented here should be used in place of the original seven-factor version. However, there remain theoretical and methodological issues, mainly around validity, to be explored through further work. The fact that the PSCQ-4 measures (organizational learning; blame culture; working conditions; and safety focus) are related to a measure of overall safety suggests that they have convergent construct validity. However, what has not been examined in this study is criterion-related validity, that is, the relationship between the PSCQ-4 measures and external markers of safety performance such as the number of complaints received or the number of medication errors reported. Such markers would be less susceptible to common method variance than was the rating of overall safety used here as a proxy criterion variable; however, they would also be potentially more difficult to measure. As a more general point with regard to validity, one argument made in favour of the seven original dimensions was that they corresponded to MaPSAF [5, 9]. In suggesting a different measurement model, this study raises the question of how faithful this new model is to MaPSAF. A cursory examination of the four factors suggests that they remain consistent with this framework; a high score on organizational learning and safety focus is consistent with the most important features of a generative safety culture, while a high-score blame culture is consistent with the main features of a pathological safety culture. While a high score on the working conditions factor does not correspond directly to a generative safety culture, it perhaps facilitates the development of such a culture. Table 6 shows how the original seven dimensions Table 6 Correspondence between the four-factor structure and the original seven dimensions from Ashcroft et al. [5, 9] Factor Dimensions... Organizational learning Blame culture Working conditions Safety focus map onto the four-factor structure. The reader will notice that only six of the dimensions are represented here; as shown in Table 3, the items that comprised team working, were removed during the factor analysis due to low factor loadings. While there still appears to be correspondence between the PSCQ-4 measures and MaPSAF, the authors suggest that empirical work is conducted to compare a pharmacy s profile of PSCQ-4 scores with its position on MaPSAF. A particular focus of such work might be on linking the PSCQ-4 measures to diagnosis and intervention on safety issues in pharmacies [19]. An alternative implication of the current findings is that MaPSAF itself could be interpreted in terms of the four dimensions; again, further empirical work would be necessary to establish whether this is the case. Finally, the sample obtained provides a reasonably broad representation of community pharmacy staff across Western Europe, which the authors considered to be sufficient for the purposes of psychometric evaluation. However, as Table 1 shows, the samples from each country are not comparable in terms of their composition, which precludes any generalizable comparisons being made between them on the basis of PSCQ scores. While it would be interesting to conduct such an exercise, either within or between countries, it is necessary to standardize the sampling frames more than was possible here. The current study, however, does offer an instrument that can be used to assessing a pharmacy s safety climate. The instrument is grounded in a theoretical rationale as well as having sound psychometric properties. In addition, it has been designed specifically for use in community pharmacies, and hence is possibly more attuned to that work setting than a generic instrument that was developed elsewhere [20]. The authors suggest further development work to confirm the relationship between PSCQ-4 and safety performance as well as to demonstrate how it can be used to inform safety improvements in pharmacies. Acknowledgements Investigating and learning from incidents Communication within the pharmacy Perception of incident causation and reporting Commitment to patient safety Staffing and management Commitment to patient safety Education and training about safety The authors would like to thank the Royal Pharmaceutical Society of Great Britain, the Royal Dutch Association for 21

7 Phipps et al. the Advancement of Pharmacy and the Ordem dos Farmacêuticos (Portugal) for their assistance with this study. Funding The Dutch data collection was made possible by a grant from the Dutch Ministry of Health. This work was also supported by Pharmakon, Danish College of Pharmacy Practice and the Danish Association of Pharmacists. References 1. Department of Health. An Organisation with a Memory. London: HMSO, Reason J. Achieving a safe culture: theory and practice. Work Stress 1998;12: Guldenmund FW. The nature of safety culture: a review of theory and research. Saf Sci 2000;34: Nieva VF, Sorra J. Safety culture assessment: a tool for improving patient safety in healthcare organizations. Qual Saf Health Care 2003;12(Suppl II):ii17 ii Ashcroft DM, Parker D. Development of the Pharmacy Safety Climate Questionnaire: a principal component analysis. Qual Saf Health Care 2009;18: Cox S, Flin R. Safety culture: philosopher s stone or man of straw? Work Stress 1998;12: Scott T, Mannion R, Davies H et al. The quantitative measurement of organizational culture in health care: a review of the available instruments. Health Serv Res 2003;38: Kirk S, Parker D, Claridge T et al. Patient safety culture in primary care: developing a theoretical framework for practical use. Qual Saf Health Care 2007;16: Ashcroft DM, Morecroft C, Parker D et al. Safety culture assessment in community pharmacy: development, face validity and feasibility of the Manchester Patient Safety Assessment Framework. Qual Saf Health Care 2005;14: Tabachnick BG, Fidell LS. Using Multivariate Statistics. 4th ed. Boston: Allyn and Bacon, Allison PD. Missing data techniques for structural equation modeling. J Abnorm Psychol 2003;112: Hu L-t, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new approaches. Struct Equ Modeling 1999;6: Karasek RA. Job demands, job decision latitude, and mental strain: implications for job redesign. Adm Sci Q 1979;24: Karasek RA. Lower health risk with increased job control amongst white collar workers. J Org Behav 1990;11: Hair JF, Black B, Babin B et al. Multivariate Data Analysis. 6th ed. Upper Saddle: Prentice-Hall, Burnham KP, Anderson DR. Model Selection and Multimodel Inference. 2nd ed. New York: Springer, Akaike H. Factor analysis and AIC. Psychometrika 1987;52: Hurvich CM, Tsai C. Regression and time series model selection in small samples. Biometrika 1989;76: Fleming M, Wentzell N. Patient Safety Culture Improvement Tool: development and guidelines for use. Healthc Q 2008;11: Waterson P, Griffiths P, Stride C et al. Psychometric properties of the Hospital Survey on Patient Safety Culture: findings from the UK. Qual Saf Health Care 2010;19: