SOCIAL-ECOLOGICAL CONSIDERATIONS IN THE EVALUATION AND IMPLEMENTATION OF ALCOHOL AND HIGHWAY SAFETY PROGRAMS

Transcription

1 SOCIAL-ECOLOGICAL CONSIDERATIONS IN THE EVALUATION AND IMPLEMENTATION OF ALCOHOL AND HIGHWAY SAFETY PROGRAMS R. P. Lillis, B.A.*, T. P. Williams, B.S.*, and W. R. Williford, M.P.H.* SYNOPSIS This paper describes an evaluation research technique which provides a comparison county for use in highway safety program evaluation. The technique takes into account a variety of county-level social factors which are measurable through readily available state data systems. Steps in development of the matching technique are outlined for application in a variety of settings. INTRODUCTION The recent rapid growth in concern about drinking and driving has led to the development of numerous alcohol and highway safety programs. These programs have evolved in response to federal and state incentives and, generally, have been implemented at the county level. Following federal and state guidelines, the vast majority of these programs have been law enforcement oriented, including special enforcement units and enhanced procedures for streamlining the prosecution of drinking drivers. Many of these programs have been accompanied by aggressive media campaigns aimed at raising the public's awareness of the drinking-driver problem and creating the perception of risk of being apprehended for drinking, and driving. Unfortunately, many localities have accepted the recommended programs without due consideration of the nature and extent of specific highway safety problems in their particular area. Often, baseline data do not exist and specific, measurable outcome goals and objectives have not been defined. Often, adequate consideration of how to determine the relationship of program activity to changes in alcohol and highway safety problem is given only after the program has been operational for some time. From the perspective of evaluation, alcohol and highway safety programs should be developed after an empirical needs assessment has been conducted, specific problem areas identified, and measurable goals set. Thus, all alcohol and highway safety programs would have clearly defined goals and * New York State, Division of Alcoholism and Alcohol Abuse, 194 Washington Avenue, Albany, New York 12210, USA. 859

2 objectives that have been formulated based on the be:st possible empirical assessment of the problem. Clients or "subjects" to be served by the program would be selected either randomly or on a matched basis, with a control group established for comparison purposes. The current situation of rapid growth of alcohol and highway safety programs presents a host of serious methodological problems for program evaluators, including: a lack of empirical baseline needs assessment data; a lack of random or matched assignment of clients; and a lack of a comparison population. In addition, the lack of specific goals for many local programs presents problems in data collection, measurement, statistical design, and interpretation of findings. In this paper, we will address specifically the issue of appropriate comparison groups for programs that have specified goals or objectives of impacting on one or more aspects of the alcohol and highway safety problem at the county level. The use of one county as a comparison or control for another county has often been used in evaluation research. In most cases, the county chosen for comparison is similar in size and population. This technique, however, is somewhat crude and, often, critical differences between experimental and control counties are overlooked. The matched control county design takes into account not only the detailed social ecology of a county but, furthermore, selects counties based on their comparability on those ecological variables that are statistically related to the major impact variable. In the case of a highway safety program that has been designed to reduce fatal crashes, for example, counties can be matched on their comparability on specific ecological factors that have been demonstrated to be significant predictors of the rate of fatal crashes. An analogy is the case of research conducted on the role of a particular substance in the development of lung cancer. In selecting a comparison group for those who have been exposed to the substance, we would attempt to match the control and the experimental subjects by age and, possibly, by selected physical characteristics. If we had information leading us to believe that the use of cigarettes and the incidence of lung cancer are correlated then we would also need to assess cigarette smoking among our experimental and control subjects and match them accordingly. METHOD In developing the match control county design in New York State, the Division of Alcoholism and Alcohol Abuse began with nearly 300 specific variables representing many 860

3 aspects of the social ecology of counties (e.g. crime rates, income level, geography). Incidence data for each of these variables were extracted from the existing information systems of numerous state agencies. Following an assessment of the reliability, validity, and overall availability of the data for each of the variables, approximately 60 variables were entered for the initial analysis prior to the development of the matching procedure. An important criterion in the selection of the specific variables was the availability of data in a consistent format not only for the counties in New York State but, whenever possible, the availability of data in a comparable form for counties throughout the United States. The major categories of variables were the following: Reported incidences of major crimes from the Uniform Crime Report. These include murder, rape, robbery, assault, burglary, larceny, and auto theft. The incidences of the major causes of death obtained from death certificates. These include death by murder, suicide, cirrhosis, diabetes, heart attack, cancer, and stroke, and infant mortality and non-motor vehicle accidental deaths. Other ecological factors including the urban population, the per capita income, the total miles of highway, employment in the county, the number of licensed outlets for alcoholic beverages, the number of state, municipal, and county police, and the number of licensed drivers in the county. Other highway safety-related variables including the incidence of arrest for speeding, reckless driving, and other major moving violations. Also, in this category the total number of fatal or injury crashes as well as damage crashes and the number of total reported crash factors in crashes other than alcohol reported. These factors are divided into human factors such as fatigue or inexperience; environmental factors such as darkness or wet roads; and vehicle factors such as faulty brakes. Details of the statistical procedures used in developing the match control county design are presented in 861

4 a separate research report which will be available soon from the Division of Alcoholism. RESULTS The results of the major steps in the development of the matched county design are: Select Measures The first step was to select an appropriate single measure for each of the ecological variables listed above. New York State comprises 62 counties which are extremely heterogenous in terms of the social-ecological variables used in this analysis. This heterogeneity provided an ideal setting for the development of the match county design. However, it presented some problems statistically in that the smallest counties often had very low or possibly no reported incidence for some variables in any given year. Other counties showed a single year with a radical increase or decline in a single variable which was otherwise constant over time. To overcome these shortcomings a 5-year geometric mean of reported incidence for each of these measures was computed. This not only provided a more conservative estimate of the variable but also smoothed the curve of 5 data points. The period studied was Data Reduction The second step in developing this procedure was a data reduction step. Numerous factor analyses were conducted on the variables. The result was the development of 5 uncorrelated factors which describe the major social-ecological factors accounting for variance between the counties in New York State. These factors are: a) A mortality factor: the incidence of diabetes, heart attack, cancer, stroke, and non-motor vehicle accidental deaths. b) A crime factor: reported murders, rapes, robberies, and auto thefts. c) A fertility factor: the number of live births, a negative loading of infant deaths and marriages. d) A highway crash factor: the total reported vehicle related factors, the total reported environmental 862

5 factors, the total non-alcohol related property damage crashes, and the total non-alcohol related human factors reported in crashes. e) A county level enforcement potential factor: the number of uniform sheriff's officers and the number of sheriff's patrol vehicles. Thereafter, we proceeded as follows; 1) The computation of county ecological factor scores. This step included multiplying the value of each county's incidence for the variables that were included in the factor analysis by the respective co-efficients produced in the factor analysis. 2) The determination of the relationship between these factors and some of the major outcome variables for local highway safety programs. Using various regression models we predicted the numbers of fatal crashes and the numbers of alcohol-related crashes in the counties from the 5 factors mentioned above. The various regression equations accounted for 40% to 72% of the variance in the major outcome measures. 3) The development of county level regression equation scores. This entailed the multiplication of each county's factor scores by the beta weights from the regression equations the result of which was that each county was given a regression equation value predicting each of the outcome measures. In other words, each county now had a single statistical value which represented the ecological factors that were most pertinent to the level of fatal crashes and one that represented each of several other major outcome variables. 4) The development of a matrix of appropriate matches for each of the counties in which evaluation was to take place. This was done by simply substracting the experimental counties' regression score from the scores of all other counties. The county or counties whose differences were the smallest were chosen as the best matches. In other words, when each county's score which predicted the number of fatal crashes was compared to each other county's, that county which had the most similar predicted value was considered to be the best social-ecological match. 863

6 DISCUSSION At this time, the matched control county procedure is being applied in several major evaluations of local programs which are intended to have county level impact on various specific outcome measures. Vie are pursuing the possibility of expanding the concept of the match control county design to apply to counties outside of New York State. This expansion will allow comparison of one or more New York counties with one of more counties from other states and allow for evaluation of programs which have been implemented statewide. In utilizing a match control county design, we recognize that the development of the factors and the regression formulas are most appropriate for the final year used in the data analysis. Utilizing a longer period of time gives greater confidence that the phenomenon being measured is actually characteristic of the county's level on any given social-ecological factor. An analysis of most of the variables used shows that the county's relative level of a factor is fairly consistent over time. However, before a post-program analysis is conducted the conditions in the selected comparison county should be studied for the possible implementation of programs similar to the experimental county or for possible significant changes in ecological factors. Likewise, the factors and the regressions should be recomputed, updating the data for the time period between the initial county selection and the post-program measurement period. CONCLUSION The development of the match control county design represents a tool which local programs can apply in measuring the impact of local alcohol and highway safety countermeasures. Although not a substitute for a well-developed needs assessment and evaluation design, it meets the needs for a powerful quasi-experimental design given today's programs. 864