MEASURING QUALITY: THE RELATIONSHIP BETWEEN DIARIES AND DIRECT OBSERVATION OF STAFF

Transcription

1 The British Journal of Developmental Disabilities Vol. 44, Part 1, JANUARY 1998, No. 86 MEASURING QUALITY: THE RELATIONSHIP BETWEEN DIARIES AND DIRECT OBSERVATION OF STAFF John Rose Introduction With a greater emphasis on service evaluation, value for money and higher standards, the measurement of quality of life is an increasingly important aspect of services for people with a learning disability. A number of authors have proposed models for conceptualising quality. In a recent review Felce and Perry (1995) portrayed quality as being represented by five domains: 1. Physical Well-being: this subsumes health, fitness and physical safety. 2. Material Well-being: this includes, finance or income, quality of the living environment and privacy, possessions, meals or food, transport, neighbourhood, security, and stability or tenure. 3. Social Wellbeing: this comprises two major dimensions of a) the quality and breadth of interpersonal relationships and b) community activities and the level of community acceptance and support. 4. Development and Activity: is concerned with the possession and use of skills in relation both to self determination and the pursuit of functional activities such as work or leisure. 5. Emotional Well-being: including, affect or mood, satisfaction, self esteem and religious faith. The model is suggested as a framework for organising the measurement of the quality of life concept. While the model implies the use of a wide range of measurement strategies, in practice the number of domains sampled is generally considerably restricted. This is due to a number of factors, such as time constraints or practical difficulties in obtaining some types of information. There has been relatively little research examining agreement across different measures. However, Perry and Felce (1995) collected information on 14 different measures of service quality. When these were grouped into a number of similar categories there was fair agreement between them. Two methods of evaluation that are concerned primarily with the development and activity sub-domain, but may provide information on others, are diaries and direct observation. Both of these methods have been used extensively, particularly in evaluating the move from residential care in hospital to the community (Emerson and Hatton, 1994). In general both methods demonstrate improvements in quality on moving from hospital to community settings. Levels of resident engagement by staff generally increase (Emerson and Hatton, 1994). Diary studies also report an increase in the use of community facilities (e.g. Bratt and Johnson, 1998; de Kock et al., 1987). However, these improvements tend to be from a very low baseline John Rose Clinical Tutor, Centre for Training in Clinical Psychology, Whitchurch Hospital, Cardiff CF4 7XB. Wales, UK. Tel: 0044 (0) Fax: 0044 (0)

2 in hospitals and there is considerable variability in the results. Other studies have had a specific focus on the comparison between direct observation and diaries (e.g. Atkinson, 1985; Joyce et al., 1989). Atkinson (1985) simply described the use of a number of methods including third party observation and the use of diaries kept by the individuals who were being observed. While there was no direct (statistical) comparison between methods, there was a clear indication that they were a potentially useful source of information on quality of life. However, she felt the techniques may have considerably different uses in practice. Another study (Joyce et al., 1989) compared direct observation of activity with the use of a structured diary. These records were kept in detail for three people but even when the results were averaged over three hour blocks, they concluded that substantial inaccuracies occurred. The results suggested that considerable caution was needed when using diaries as a source of evaluation data. However, diaries are a relatively simple form of measurement that are considerably less time consuming than direct observation; they can also be far less obtrusive. While there is concern with the use of diaries as a source of evaluation data, their convenience makes them attractive as an indicator of service quality. It is with this in mind that the comparisons in this study were made. The study simply consisted of a comparison between the results of observations made on staff who work in community residential settings, with diary recordings made over a similar time period. It was hypothesised that increases in observed levels of interaction by staff would correlate with an increase in diary recordings of programmes and activities. Method Settings The study was conducted in a range of community residences. Initial approaches were made through management. One community unit and four group homes were identified for study. The community unit consisted of three adjacent bungalows (two with six and one with seven full time residents). The four group homes were ordinary houses all with four full time residents. All residences were managed as part of the same NHS Trust. Residents and Staff Residents ranged in age from years (mean 51). In total there were 16 women and 19 men. They were assessed using the ABS (Adaptive Behaviour Scale) part 1 (Nihira et al., 1974) and on the Behaviour Problems Inventory (Rojahan et al., 1989). A total mean score of 119 (range ) was recorded on the ABS. The mean number of types of challenging behaviour recorded per resident on the Behaviour Problems Inventory was 4.83 in the previous month. Only direct care staff were included in this research of which there were forty four in the facilities, 36 were women. They ranged in age from 18 to 60. Observations Staff were observed using Psion Organisers (Mark 2) programmed for a 30 second momentary time sample. Software developed for resident observation (Beasley, Hewson and Mansell, 1989) was recoded, so that it could be used to observe staff. In 31

3 total, three observers were used to collect the data. The observation schedule required a judgement as to who the member of staff being observed was interacting with (if anyone), the content of the interaction, the type of the interaction, and what activity the member of staff was involved in. Staff:client numbers were recorded every five minutes. TABLE I details the various observational categories. Individual staff were observed for 90 minute periods. Shift patterns were similar in the services, consisting of seven and a half hour morning and afternoon shifts, with no formal break but a changeover between shifts. A general plan was followed for each house or bungalow, whereby all of the times between 7.30 a.m. when staff started the morning shift, up until 10 p.m. when staff finished the evening shift, they would be observed once on a weekday (i.e. 9 observations). Three additional observations were made at weekends to try and provide a representative cross section of a week s work. Out of a total staff group of 44, only 4 staff refused to participate in the observations. Unfortunately, two staff from one group home refused to participate; this house also had the smallest compliment of staff (4), due to staff vacancies and sickness, working regularly at the time of the study. As only two people were willing to be observed, it was not possible to obtain the full number of observations in this house. This meant that 12 observations were conducted in six of the houses and community unit bungalows (18 hours each) but in one house only six one and a half hour observations were made (9 hours). In total there were 117 hours of observations. For a more complete description of the observational protocol see Rose et al. (1994). Reliability Two observers were present for fourteen and a half hours (nine complete and one partial observation) representing 12.4% of observations. Inter-rater reliabilitys were calculated for individual observation sessions using a programme that calculates both percentage agreement and Kappa for each variable (Martin, 1992). When data was combined across all sessions relating to the settings, Kappa values were reasonable for all codes, ranging from 0.61 to 1.0. All percentage agreements were in excess of 88%. Diary Recordings All of the residential units taking part in this research required staff to keep a diary detailing some aspects of resident activity on a monthly basis as part of a quality monitoring programme. Information was kept on Day to Day Activities, defined as any activity outside of the house, e.g. visits to shops, church or day centre and sessions with the occupational therapist at home. Information was also recorded on Personal Plans; these were something important for one particular client. Personal plans included skill development programmes or personalised leisure activities. There were no precise definitions for this category of activity, but all staff had to agree on the criteria for recording. Individual criteria were then agreed between residential staff and the researchers responsible for data collection. Information was written on a daily basis onto a specially designed activity record (Thomas and Ponton, 1989). Data corresponding to the three month period most closely associated with the period in 32

4 which observations were made was collected. Results Means were calculated for each observational category for each observation. Total numbers of day to day activities and personal plans were calculated over the three months for each house or bungalow. The number of daily activities and plans per month were then calculated and finally the number per resident per month by dividing the monthly total by number of residents in the facilities. These two measures were then correlated (Pearson, 2 tailed tests) in order to explore the relationship between them. The results are presented in TABLE I with significant correlations (greater than 5% significance) being shown in bold type. A number of correlations are shown, including negative correlations between a lack of interaction and personal plans and day activities (fewer plans/activities are associated with higher levels of no interaction). Positive correlations are noted between staff contact with residents and others from outside with personal plans and day activities (more plans/activities are associated with higher levels of contact with these groups). Similarly, more personal plans and day activities are associated with an increase in work related interactions, neutral interaction and more observed formal educational programmes. There were also associations between more contact with visiting professionals, observed personal care and personal plans. A negative correlation was also found between day activities with personal time (defined as the number of breaks staff took from work while being observed). That is, the more personal time observed, the fewer day activities recorded. There was a negative correlation between the number of clients observed and the number of personal plans and activities performed (the larger the number of clients the fewer plans and activities). However, the relationship between staff numbers with activities and plans was not significant. Discussion Probably the most important finding of this study is the strong degree of correlation between diary records and staff behaviour on a number of key quality indices. For example, the positive correlation between increased resident interaction (in general, particularly neutral interaction and work related interactions) with personal plans and day activities. The positive correlation between formal educational programmes and more plans and activities is pleasing, but not surprising as formal education is likely to include many personal plans and possibly some day activities. An increase in all of these measures have been considered to be synonymous with improved quality. The negative correlations between the diary recordings and lack of interaction are simply representing the opposite relationship: the relationship between a lack of interaction and fewer personal plans and day activities. However, the lack of any correlation between assistance and diary recordings is of some concern as this is considered to be a primary indicator of quality by some authors (Felce, 1994). Other relationships are of interest, such as that between professional and other visitors with some increased levels of diary recordings. This may be symptomatic of houses that seek external input generally providing a better quality service. Alternatively, these external contacts may facilitate 33

5 TABLE I Correlations of clients personal plans and day activities with observations of staff behaviour in group homes and community units. Social Behaviour (communication with) Staff Behaviour Personal Plans Day Activities (Pearson Correlation Coefficient/Probability) None -.395/ /.000 Resident.403/ /.001 Peer -.174/ /.228 Manager -.039/ /.484 Visiting Professionals.298/ /.052 Residents Families, Friends, Advocates and others outside.323/ /.004 Observer, Other -.068/ /.533 Content of Interaction None -.402/ /.003 Work related.422/ /.001 General Social.023/ /.795 Interaction Type None -.397/ /.003 Positive.031/ /.352 Negative -.046/ /.651 Neutral.404/ /.001 Assistance.004/ /.751 Type of Work/Activity Personal Care.229/ /.179 Housekeeping (Domestic, Non food related).013/ /.232 Cooking -.173/ /.291 Shopping.056/ /.911 Leisure.143/ /.081 Records/Administration/ Managing others -.162/ /.081 Being Managed/Meetings/ Personal Development -.174/ /.159 Personal Time -.207/ /.046 Formal Education.313/ /.005 Other.002/ /.138 Staff/Client Numbers Staff : Client Ratio.234/ /.005 Staff No..002/ /.318 Client No / /

6 more activities and plans. The relationship between personal time (time spent on informal breaks from work) and an apparent reduction in the number of day activities is perhaps predictable. However, it should be noted that these staff were expected to work long hours without an official break. There was also no indication that time spent chatting with other staff about non work matters (general social) had any effect on the numbers of personal plans and day activities recorded. The correlation between personal care and personal plans may have a number of explanations. For example, it may be that many plans were set in the area of personal care. Alternatively, there may be some sort of effect of resident ability influencing these results. The lack of any strong relationship between other activities staff perform with diary recordings is in general not surprising as simply being involved in activity does not tell us anything about resident participation. However, a positive relationship with externally oriented activities, such as shopping, may have been considered likely. The relationship between decreased client numbers with more recorded personal plans and day activities suggests reducing client numbers facilitates these events. The lack of a relationship with staff numbers suggests that increasing numbers of staff while maintaining the size of the client group is not as likely to increase the number of personal plans and activities recorded by staff. This finding is consistent with previous research which has found only minimal improvements in staff performance and client behaviour arising from increasing staff numbers in a client group of a given size (Felce, et al., 1991). A picture emerges from these data of a clear relationship between diary and observational measures, even though the observational data were collected over a much shorter time than the diary data. The relationship suggests a degree of consistency between both measures. The fact that data was collected only from community residences would also support the utility of the measures as the extreme variability seen between hospital and community residences was not present in this sample. There are clearly some problems with this study, for example, the use of multiple correlations means that false positive correlations may occur (type 1 error). The lack of direct comparability between measures could also be seen as a weakness in the research design. However, it could also be argued that this demonstrates a certain robustness of the data. Other concerns may centre on the fact that the measures reported here were focused on staff behaviour, calling into question the direct relevance of the data for residents. There may also be other confounding factors such as differences in resident ability between residences. This study has shown a reasonable degree of correlation between diaries and direct observational measures on some key variables. It would therefore be appropriate to conclude that they are to some extent measuring similar things and could both provide a reasonable indication of quality in the domains they sample. These data would tend to support the use of diaries as a useful and pragmatic form of data collection. Therefore, both types of data should be considered for future quality monitoring and evaluation exercises. Clearly, neither observation or diaries can provide a complete picture of a service; to obtain this, other measures would be required. However, it is likely that both of these sources of data will continue to provide an important source of information in monitoring and evaluating learning disability services. 35

7 Summary This study examined the relationship between two frequently used measures of quality: direct observation of staff behaviour and diaries of client activity in community residential settings for people with learning disabilities. Various elements of staff behaviour were observed including who they interacted with, the type and content of interaction and what activities they were engaged in. Routine daily diaries were kept of resident activities (principally outside of the house but including therapy sessions in the house) and personal plans (something chosen as having particular relevance to an individual) over a three month period. These measures were correlated with each other. A range of relationships were found including positive correlations between increased numbers of both day activities and personal plans with observed staff contact with residents; formal educational programmes and contact with others from outside the home. A positive correlation was also found between an increased staff : client ratio and personal plans/day activities. However, this relationship was largely accounted for by reducing client numbers. Changes in staff numbers seemed to have little impact. The relationship between diaries and direct observation and their potential (dis)advantages are discussed. It is concluded that both diaries and observation can be potentially useful sources of information. Acknowledgements Many thanks to Maria Thomas and Sue Ponton of Oxfordshire Learning Disabilities (NHS) Trust for providing the diary data used in this research. Also, thanks to the staff who collected the data and cooperated with the observations so willingly. References Atkinson, D. (1985). The Use of Participant Observation and Respondent Diaries in a Study of Ordinary Living. British Journal of Mental Subnormality, 31, Beasley, F., Hewson, S. and Mansell, J. (1989). MTS: Handbook for observers. Canterbury: Centre for Applied Psychology of Social Care, University of Kent. Bratt, A. and Johnston, R. (1988). Changes in lifestyle for young adults with profound handicaps following discharge from hospital care into a second generation housing project. Mental Handicap Research, 1, de Kock, U., Felce, D., Saxby, H. and Thomas, M. (1987). Staff turnover in a small home service: Study of facilities for adults with severe and profound mental handicaps. Mental Handicap, 15, Emerson, E. and Hatton, C. (1994). Moving out: Relocation from hospital to the community. London: HMSO. Felce, D. (1994). The Quality of Support for Ordinary Living: Staff:Resident Interactions and Resident Activity. In: J. Mansell and K. Ericsson (Eds.). The dissolution of Institutions: An International Perspective. London: Chapman and Hall. Felce, D. and Perry, J. (1995). Quality of life: Its definition and measurement. Research in Developmental Disabilities, 16, Felce, D., Repp, A., Thomas, M., Ager, A. and Blunden, R. (1991). The Relationship of Staff:Client Ratios, Interactions and Residential Placement. Research in Developmental Disabilities, 12, Joyce, T., Mansell, J. and Grey, H. (1989). Evaluating service quality: A comparison of diaries and direct observation. Mental Handicap Research, 2, Martin, N. (1992). M.T.S. Reliability Programme. Unpublished. Available from the author, 36

8 Bassetts Resource Centre, Acorn Way, Starts Hill Road, Farnborough, Kent, UK. Nihira, K., Foster, R., Shellhaus, M. and Leland, H. (1974). AAMD Adaptive Behaviour Scale. Washington: American Association of Mental Deficiency. Perry, J. and Felce, D. (1995). Objective Assessments of Quality of Life: How Much do they Agree with Each Other? Journal of Community and Applied Psychology, 5, Rojahan, J. R., Polster, L. M., Mulide, I. A. and Wisniewski, J. J. (1989). Reliability of behaviour problem inventory. Journal of the Multihandicapped Person, 2, Rose, J., Mullan, E. and Fletcher, B. (1994). An Examination of the Relationship Between Staff Behaviour and Stress Levels in Residential Care. Mental Handicap Research, 7, Thomas, M. and Ponton, S. (1989). Activity Record. Witney: Windrush Press. 37