Should the Audiometric Database Analysis Method (Draft ANSI S ) for Evaluating the Effectiveness of Hearing Conservation Programs

Transcription

1 J Am Acad Audiol 6 : (1995) Should the Audiometric Database Analysis Method (Draft ANSI S ) for Evaluating the Effectiveness of Hearing Conservation Programs Be Accepted as a US National Standard? Tilahun Adera* Gail M. Gullickson* Thomas Helfert Leming Wang* John W. Gardner* Abstract The value of audiometric database analysis (ADBA) procedures for evaluating the effectiveness of hearing conservation programs (HCPs) was studied in a population of 82,195 workers. We rated the HCP using four ADBA procedures. Results showed that the HCP ratings ran the gamut from "unacceptable" to "acceptable" from year to year and from procedure to procedure. A mere 7 percent of the total population was eligible for ADBA analysis. This drastically reduced sample size precludes subgroup analyses of even a large cohort and likely could not be applied to the HCPs of most small- and medium-sized businesses. The low levels of agreement observed (weighted kappa = 15%-25%) between the ADBA procedures reflect poor validity. Generalizability of these results would be inappropriate when so few nonrandomly selected individuals contribute to the evaluation. In view of these problems, acceptance of the ADBA method as a national standard may not be warranted. Key Words: Audiometric database analysis, evaluation of hearing conservation programs S ince 1983, employers have been required by law to provide hearing protection for their employees exposed to noise levels that equal or exceed an 8-hour time-weighted average sound level of 85 db measured on the A scale (OSHA, 1983). This requirement resulted in the establishment of numerous hearing conservation programs (HCPs) throughout industry. Unfortunately, the performances of these programs in preventing noise-induced hearing loss have seldom been documented. Part of the "The Uniformed Services University of the Health Sciences, Department of Preventive Medicine and Biometrics, Bethesda, Maryland ; tthe US Army Environmental Hygiene Agency, Aberdeen Proving Ground, Maryland Reprint requests : Tilahun Adera, Medical College of Virginia, Virginia Commonwealth University, 1008 East Clay St., P.O. Box , Richmond, VA reason for this lack has been the unavailability of accurate and reproducible scientific methods to assess the effectiveness of HCPs. Recently, however, audiometric database analysis (ADBA) procedures to evaluate the effectiveness of HCPs have been developed and proposed by the American National Standards Institute Working Group (ANSI S12/WG12) (ANSI, 1991). The purpose of the draft standard is "to define objective procedures for evaluating HCP effectiveness in preventing occupational noiseinduced hearing loss in a noise-exposed population" (p. 4). According to the draft standard, ADBA can be used to improve HCPs by identifying potential problem areas in the HCP before employees develop significant hearing loss. In addition, the draft standard makes the claim that the ADBA method can be used to motivate and increase awareness of the importance of the HCP and provide tangible benefits for both the noise-exposed employee and the employer. The

2 Hearing Conservation Programs Evaluation/Adera et al stated goal of the draft standard is to promote the prevention of noise-induced hearing loss through a systematic assessment of the effectiveness of HCPs. Despite this worthy goal, the accuracy and reproducibility of the ADBA method in evaluating the effectiveness of HCPs has not been well established. This issue has important implications since the method has recently appeared in various publications (Royster et al, 1980 ; Melnick, 1984 ; Royster and Royster, 1986a, b, 1988, 1990a, b, c, 1991 ; Suter, 1989 ; Simpson et al, 1993) and is currently being considered for adoption as a national standard (ANSI, 1991). The ANSI Working Group has invited comments and criticisms from users and researchers, stating that the "Draft Standard has been approved by Accredited Standards Committee S12 for trial use, comment, and criticism, and has been published in order to obtain those comments which will occur as a result of its use. When sufficient time has elapsed for trial use of the Draft Standard and subsequent receipt of comments, the Draft Standard will be amended as needed, and revised text will be submitted for approval as an American National Standard" (ANSI, 1991). Following this request, we present an analysis and a critique of the ADBA method resulting from application to audiometric data collected from civilian employees in the US Army during 1968 to To our knowledge, this is the largest single data set (N = 82,195) reported to date using ADBA procedures. We examined the ADBA method and sought to answer three questions. The first is, "Is it practical to use the ADBA procedures for evaluating the HCPs of small- and medium-sized businesses or subgroups of a large population?" In other words, "Does each of these organizations have enough noise-exposed workers to enable them to use the ADBA procedures?" This issue was raised because of the method's highly restrictive selection requirement that includes annual audiometry with intervals between tests not to exceed 18 months. Additionally, it is required that all members of the cohort must have their first test in a given year with a minimum of four consecutive annual tests (ANSI, 1991). Previous research has shown that only a small fraction of a set of audiometric data met the ADBA criteria for analysis (Adera et al, 1993a). This information may help small- and medium-sized organizations to decide if they have the necessary sample size to apply the ADBA procedures. The second question is, "Are the results of the ADBA procedures generalizable to the entire study population?" The answer to this question would be "yes" only if the sample chosen, using the ADBA selection criteria, is representative of the study population. The third question is, "How consistent are the results from the various ADBA procedures?" In other words, "What is the extent to which ADBA procedures agree in rating an HCP?" This question is asked because of our previous research that suggested extreme rating variability among procedures (Adera et al, 1993a). Subject Selection METHODS The subject selection process for application of the ADBA method is shown in Figure 1. The original study population consisted of 82,195 civilian employees of the US Army who had a total of 258,472 audiograms during 1968 to The majority of employees worked within a single major army command. Although the study population in the HCP to be evaluated was considerably large, only 6.8 percent (5576/82,195) of this population and 28.5 percent (5576/19,397) of the population with 3 years of follow-up were eligible to be included in the ADBA method of analysis. The ADBA procedures were applied to two cohorts of employees : those who met the ADBA criteria of four consecutive annual tests (cohort-a4, N = 1193), and those who met the criteria of eight consecutive annual tests (cohort- A8, N = 260). According to ADBA criteria, we used four steps in selecting subjects for entry into cohort- A4. First, we selected all reference and annual audiograms of each worker. Second, if more than one test was recorded in a calendar year, we selected the first test. Third, we set the interval between adjacent tests to be at least 6 months and not more than 18 months. Fourth, each subject had to have at least four consecutive tests in successive calendar years, with the first test completed in a specific year. We found that 1982 was the year in which the largest number of employees had their first test, and so all subjects were required to have their first test in Application of these requirements for entry into cohort-a4 resulted in 1193 subjects (1094 males and 99 females). Members of cohort-a4 who had at least eight consecutive tests formed cohort-a8 (i.e., 246 males and 14 females). Thus, cohort-a8 is a subset of cohort-a4. Within each cohort, the HCP was rated using four ADBA procedures of percent worse sequential (%Ws),

3 Journal of the American Academy of Audiology/Volume 6, Number 4, July 1995 Males (N =1094) Males (N=246) Study Popolation Study Period : (N=82,195) Three Years of Follow-up. (N= 19,397) Three Years of Follow-up + Four Consecutive Tests (N=5576) Cohort-A4 Follow-up : Three Years Follow-up + Four I Consecutive Tests + First Test in 1982 (N=1193) Females (N=99) Cohort-AS Follow-up : Three Years of Follow-up + 8 I Consecutive Test + First Test in 1982 (N=260) Females (N=14) Figure 1 Subject Selection Process for Application of Audiometric Database Analysis (Draft ANSI S ). percent better or worse sequential (%BWs), standard deviation applied to single test frequencies (STDVs), and standard deviation applied to averages of test frequencies (STDVAs). Hearing Conservation Program The HCP of the US Army has been fully described elsewhere (Department of Army [DA], 1991 ; Hearing Evaluation Automated Registry System [HEARS], 1991) and only a brief summary follows. Civilian employees were enrolled in the army-wide HCP at many different installations throughout the country if they were exposed to (a) steady-state noise of 85 dba or greater in the audible range, up to 16,000 Hz, regardless of duration ; (b) impulse noise of 140 peak db or greater; (c) airborne high-frequency or ultrasonic noise, despite duration, in any of the 1/-octave bands exceeding the corresponding value listed (DA, 1991). All noise-exposed civilian personnel were required to receive reference, 90-day, annual, and termination audiograms. Reference audiograms for new employees were done by 30 days after initial exposure to hazardous noise, and termination audiograms were conducted at least 1 week before the employee's termination. Audiometric tests administered were pure-tone, air-conduction, hearing threshold examinations w ith tests including 500, 1000, 2000, 3000, 4000, and 6000 Hz. Tests at each frequency were taken separately for each ear. Tests were repeated if there was (a) a 50-dB difference between adjacent test frequencies ; (b) a hearing threshold level greater than 30 db at 500 Hz ; (c) a +20-dB hearing shift at 1000 to 4000 Hz ; and (d) any threshold greater than 90 db. Noise surveys have been conducted by an industrial hygienist in all potentially noise-hazardous areas at least once a year. The sound level meter used to survey noise levels met or exceeded the requirements for a type 2 sound level meter per ANSI Standard S (ANSI, 1983). All personnel working in noise-hazardous areas were required to have hearing protection devices with them at all times. Hearing protection devices include earplugs, noise muffs, ear canal caps, noise-attenuating helmets, or a combination of these. They were provided at no charge to all personnel working in noise-hazardous areas. Personnel were permitted to choose the type of protection device they desired unless their selection was medically contraindicated or inappropriate for a particular noise-hazardous area. The type of protection (single or combination protectors) required at different levels of noise and the maximum allowable duration of exposure have been specified elsewhere (DA, 1991). Data Analysis The HCP was evaluated by rating various test comparisons using ADBA procedures and estimating the reproducibility of these ratings. The ADBA procedures were based on comparisons of sequential audiograms for measuring year-to-year variability in hearing threshold levels (ANSI, 1991). These procedures were %Ws, %BWs, STDVs, STDVAs and are discussed below. Percent Worse Sequential The %Ws procedure computes the proportion of subjects per 100 whose hearing worsened by > 15 db between any two sequential audiograms at any test frequency (0.5, 1, 2, 3, 4, or 6 khz) in either ear. To compute the %Ws procedure, the difference between two sequential audiograms was calculated for each pair of sequential audiograms (e.g., test 2 - test 1, test 3 - test 2, etc.) at each of the test frequencies (0.5, 1, 2, 3, 4, or 6 khz) for both ears. A total of 72 sequential test differences were calculated for a cohort followed

4 Hearing Conservation Programs Evaluation/Adera et al for approximately 7 years (i.e., six test comparisons, six test frequencies, and two ears : 6 X 6 X 2 = 72 ; note that the draft standard does not provide a rating for test 4-5 comparisons). A person was considered to have worse hearing if the differences between any sequential tests were at least 15 db. Percent Better or Worse Sequential The %BWs procedure computes the proportion of subjects per 100 whose hearing level changed toward better or worse hearing by > 15 db between two sequential audiograms at any test frequency (0.5, 1, 2, 3, 4, or 6 khz) in either ear. To compute the %BWs procedure, the difference between two sequential audiograms was calculated for each pair of sequential audiograms at each of the test frequencies (0.5, 1, 2, 3, 4, or 6 khz) for both ears. A total of 36 sequential test differences were calculated for a cohort followed for 7 years (i.e., three test comparisons, six test frequencies, and two ears : 3 X 6 X 2 = 36). A person was considered to have a change in hearing if any of the differences between sequential tests were > 15 db or < -15 db. Standard Deviation Applied to Single Test Frequencies The STDVs procedure computes the standard deviation of the average binaural differences at each audiometric test frequency. The following steps were taken in computing the STDVs : first, we calculated the binaural means of the hearing threshold levels (HTLs) (i.e., the mean HTLs of the left and right ears) at each audiometric test frequency for each worker on each test number. Second, we computed the HTL differences of the binaural means between two sequential tests at each test frequency. Finally, we computed the standard deviation for the HTL differences across all employees on each sequential test comparison. Standard Deviation Applied to Averages of Test Frequencies The STDVAs procedure computes the standard deviation of the binaural differences at each group of low (0.5, 1, 2, 3 khz), mid (2, 3, 4 khz), and high (3, 4, 6 khz) audiometric frequency averages. We took the following steps in computing the STDVAs : first, we calculated the binaural means of the hearing threshold levels (HTLs) at each audiometric test frequency for each worker on each test number. Second, we calculated the mean HTL of the binaural means on each test at each group of the following frequencies : 0.5, 1, 2, 3 khz ; 2, 3, 4 khz; and 3, 4, 6 khz. Third, we computed the HTL differences of the above group means between two sequential tests. Finally, we calculated the standard deviation of these differences at each group of averaged frequencies on each sequential test comparison across all employees. We then compared the results from all of the procedures above to tables of criterion ranges recommended by the Draft ANSI (ANSI, 1991). By using these recommended ranges, we classified the effectiveness of the HCP as "acceptable," "marginal," or "unacceptable." Reliability To quantify the degree of agreement between procedures, we used the ratings assigned by each ADBA procedure to each test comparison. We assessed the degree of agreement by calculating a weighted kappa coefficient (kw) using r by c tables (Fleiss, 1981). The weighted kappa coefficient calculates the proportion of agreement between procedures over and above that expected by chance, that is, it corrects for the chance of agreement that would be expected to occur if two procedures were totally unrelated. RESULTS Cohort Description: Distribution by Sociodemographic Factors The percent distribution of sociodemographic factors including gender, age at entry, race, and education is presented for the entire study population and the selected two cohorts : cohort-a4 and cohort-a8 (see Table 1). The ADBAmethod selected fewer women in both cohorts (cohort-a4 : 5.4%; cohort-a8 : 8.2%) compared with the total population (13.4%). In addition, workers under the age of 25 (age was calculated at the time of entry into the cohort) were under-represented in both cohorts (total population : 6.6% vs cohort- A4 : 3.5% ; cohort-a8 : 1.9%), and both cohorts had a smaller percentage of those over the age of 54 (total population : 4.2% ; cohort-a4 : 1.1% ; cohort-a8 : 0.8%). The proportion of blacks and Hispanics declined in the two cohorts compared to the total population. This may suggest that minorities were less likely to stay in the HCP or to get annual audiograms than whites. The

5 Journal of the American Academy of Audiology/Volume 6, Number 4, July 1995 Table 1 Percent Distribution of Sociodemographic Factors in Three Groups of Workers. Demographic Total Pop Factor (N = 82,195) Cohort-A4 (N=1193) Cohort-A8 (N=260) Gender (N = 82,195) Males Females Age at entry (N = 67,114)- < Race (N = 77,511)- White, non-hispanic Black, non-hispanic Hispanic Other Education (N = 77,511)` Elementary High school Post high school/ college Post college ` Subjects with missing values are excluded. distribution by education was essentially similar in all groups. Audiometric Database Analysis of Cohort-A4 According to the requirements of the Draft ANSI S12.13 (ANSI, 1991), cohort-a4 consisted of employees who had their first test in the same year and had at least four consecutive annual tests. Tables 2 and 3 show scores and HCP ratings from the %Ws and the two standard deviation procedures in cohort-a4 (men = 1094, women = 99). For both men and women, the %Ws procedure rated the HCP as "unacceptable" in two of three test comparisons and "marginal" on the third comparison (Table 2). The mean of the %Ws values over all test comparisons was 42.9 percent for males and 36.4 percent for females. Thus, approximately 43 percent of the men and 36 percent of the women in this cohort had worse hearing, showing an "unacceptable" rating in both sexes. Based on data for men, the STDVs procedure rated the HCP as "acceptable" in 10 of 18 (55.6%) test comparisons and "unacceptable" in 7 of 18 (38.9%) test comparisons, with the remainder (5.5%) classified as "marginal" (Table 3). The corresponding percentages for women were 50 percent (9/18), 44.4 percent (8/18), and 5.6 percent (1/18). Using data for males, the STD- VAs procedure rated the HCP as "acceptable" in 67.7 percent (6/9) of test comparisons and "unacceptable" in 33.3 percent (3/9) of test comparisons. The corresponding figures for females were 44.4 percent (4/9) and 33.3 percent (3/9). From these figures, it would appear that the HCP is likely to be rated slightly better based on data from men than women. In addition, while the %Ws procedure rated the HCP in the "unacceptable" range for both men and women, the STDVs and STDVAs procedures rated it in the "marginal" range. Audiometric Database Analysis of Cohort-A8 Members of cohort-a8 had received a minimum of eight consecutive annual tests over a period of approximately 7 years (a reference test Table 2 %Ws Scores with Associated HCP s and Marginal Criterion Ranges by Test Comparison for Cohort-A4 Male and Female Department of the Army Civilian Employees during Tests Compared Sample Size Male Female Male Ws Scores Marginal HCP t Female Range" Male Female 1 to to to U U M M U U Males : mean of %Ws = 42.9 (U) ; females : mean of %Ws = 36.4 (U). " Draft American National Standard (ANSI S ) recommended criterion range for rating HCP effectiveness. Acceptable : if the value of %Ws is below the lower limit of the marginal range, marginal : if the value of the %Ws is within the marginal range, unacceptable : if the value of %Ws is above the upper limit of the marginal range. 'U = unacceptable, M = marginal, A = acceptable. 306

6 Hearing Conservation Programs Evaluation/Adera et al Table 3 Scores from STDVs and STDVAs for Cohort-A4 Male and Female Department of the Army Civilian Employees during Tests 1 to 2 (HCP t) Tests 2to3 (HCP ) Tests 3to4 (HCP ) Frequency (khz) Marginal Range' Male Female Male Female Male Female STDVs (U) 11.7 (U) 7.3 (U) 11.0 (U) 5.6 (A) 9.0 (U) (U) 9.4 (U) 5.3 (A) 6.1 (M) 4.2 (A) 5.3 (A) (U) 10.3 (U) 4.6 (A) 5.4 (A) 4.4 (A) 5'.6 (A) (U) 17.8 (U) 6.1 (A) 4.5 (A) 5.5 (A) 4.6 (A) (U) 21.5 (U) 6.8 (A) 4.9 (A) 6.3 (A) 4.7 (A) (U) 22.1 (U) 9.2 (M) 7.6 (A) 8.9 (A) 8.3 (A) STDVAs 0.5, 1, 2, (U) 8.7 (U) 4.4 (A) 5.6 (M) 3.6 (A) 5.1 (M) 2, 3, (U) 14.4 (U) 4.8 (A) 4.1 (A) 4.4 (A) 4.2 (A) 3, 4, (U) 18.5 (U) 5.7 (A) 4.1 (A) 5.2 (A) 4.4 (A) Male : N = 1094, female : N = 99. * Draft American National Standard (ANSI S ) recommended criterion range for rating HCP effectiveness. Acceptable : if the value of standard deviation is below the lower limit of the marginal range, marginal : if the value of the standard deviation is within the marginal range shown ; unacceptable : if the value of standard deviation is above the upper limit of the marginal range. 'HCP rating : U = unacceptable, M = marginal, A = acceptable. and seven annual tests). Although there was a total of 260 employees (246 men, 14 women) in this cohort, the analysis was limited to men only as the eligible number of women was less than the ADBA minimum sample size requirement of 30. Table 4 shows the results of %Ws and %BWs procedures for the 246 male employees. While the HCP was rated between "unacceptable" and "marginal" at different test comparisons by the %Ws procedure, the %BWs procedure consistently rated it as "unacceptable." The means of %Ws and %BWs for all test comparisons were 36.5 percent and 51.1 percent, respectively, classifying the HCP in the "unacceptable" ranges. The STDVs procedure rated the HCP as "acceptable" in 55.6 percent (20/36), "marginal" in 22.2 percent (8/36), and "unacceptable" in 22.2 percent (8/36) of test comparisons (Table 5). Similarly, the STDVAs procedure rated the HCP as "acceptable" in 66.7 percent (12/18), "marginal" in 16.7 percent (3/18), and "unacceptable" in 16.7 percent (3/18) of test comparisons (Table 5). Thus, while the %Ws and the %BWs procedures rated the HCP toward the "unacceptable" range, the STDVs and STDVAs procedures rated it toward the "acceptable" range. Table 4 %Ws and %BWs Scores with Associated HCP s and Marginal Criterion Ranges by Test Comparison for Cohort-A8 Male Department of the Army Civilian Employees during %Ws Procedure %BWs Procedure Tests Compared ADBA Score Marginal Range' HCP t ADBA Score Marginal Range HCP 1 to U 2 to M 3 to U 5 to M U 6 to U U 7 to U U N = 246. Draft American National Standard (ANSI S ) recommended criterion range for rating HCP effectiveness. Acceptable : if the value of %Ws or %BWs is below the lower limit of the marginal range, marginal : if the value of the % Ws or %BWs is within the marginal range ; unacceptable : if the value of %Ws or %BWs is above the upper limit of the marginal range. 'U = unacceptable, M = marginal, A = acceptable.

7 Journal of the American Academy of Audiology/Volume 6, Number 4, July 1995 Table 5 Scores from STDVs and STDVAs for Cohort-A8 Male Department of the Army Civilian Employees during Tests 1 to 2 Tests 2to3 Tests 3to4 Tests 5 to 6 Tests 6 to 7 Tests 7 to 8 Frequency (khz) Marginal Range' SD HCP t Marginal Range STDVs U 6.0 M 5.5 A U 8.4 U 5.9 M U 4.7 A 4.0 A A 5.2 M 5.4 M U 4.3 A 3.7 A A 4.7 A 5.7 M U 4.8 A 4.8 A A 5.1 A 6.6 M U 5.8 A 5.3 A A 5.8 A 6.4 A U 8.2 A 7.4 A A 8.2 M 8.7 M STDVAs 0.5, 1, 2, U 3.8 A 3.4 A M 4.5 M 4.6 M 2, 3, U 4.1 A 3.7 A A 4.1 A 4.9 A 3, 4, U 4.7 A 4.1 A A 4.4 A 5.1 A N = 246. Draft American National Standard (ANSI S ) recommended criterion range for rating HCP effectiveness. Acceptable : if the standard deviation value is below the lower limit of the marginal range ; marginal : if the standard deviation value is within the marginal range ; unacceptable : if the standard deviation value is above the upper limit of the marginal range. thcp rating : U = unacceptable, M = marginal, A = acceptable. Reliability Data for the reliability of hearing HCP ratings are shown in Table 6. The chance corrected percentage of agreement between STDVs and STDVAs was quite high (kw = 90%). This is to be expected, however, as both methods are based on the standard deviation statistic. In contrast, low levels of agreements were observed when the %Ws procedure was compared to STDVs (kw = 25%) and STDVAs (kw = 15%) procedures. U DISCUSSION sing audiometric data collected from 82,195 Department of the Army civilian employees during 1968 to 1992, we studied the value of ADBA procedures in evaluating the effectiveness of a HCP. The ADBA procedures were applied to two cohorts of employees : cohort-a4 and cohort-a8. Within each cohort, the HCP was rated using four procedures. Both cohorts were predominantly white males under the age of 45. Applications.of the ADBA procedures raised questions regarding the practicality, generalizability, and reproducibility of these procedures and are discussed below. Practicality : Problem of Sample Size Sample size problems associated with using ADBA procedures appear to be largely due to the highly restrictive selection requirement that includes annual audiometry with intervals between tests not to exceed 18 months. Additionally, it is required that all members of the cohort must have their first test in a given year with a minimum of four consecutive annual tests (ANSI, 1991). These data restrictions seem to rapidly reduce the sample size necessary for program evaluation. Table 6 Agreements between ADBA Procedures in the Performance of a Hearing Conservation Program Procedure U M S TDVs' A Total U S TDVA st M A Total %WS U M A Total STDVAs$ U M A Total U = unacceptable, M = marginal, A = acceptable, STDVs = standard deviation for single frequencies ; STDVAs = standard deviation for averaged frequencies ; WS = percent worse sequential. % BWs was not included because of insufficient number of test comparisons. `kw = 0,250;tkw = 0.154,$kw =

8 Hearing Conservation Programs Evaluation/Adera et al Further, the problem of shrinking sample size can be illustrated by considering the number of exclusions at various stages of the data selection process. Among 82,195 workers, 19,397 had 3 years of follow-up; of these, 5576 had 3 years of follow-up with four consecutive annual tests ; of these, 1193 were eligible for entry into cohort-a4 because they had 3 years of follow-up with four consecutive annual tests and had their first test in 1982 (chosen because the largest number of subjects had their first test in this year) ; of these, only 260 men had 7 years of follow-up and were eligible for entry into cohort- A8. It should be noted that although there were 10,374 women in the original population, they were all excluded from entry into cohort-a8 because the number of women left after applications of the ADBA selection criteria was too small (N = 14) to meet the minimum sample size requirement of 30. These findings of rapidly diminishing sample size are consistent with our previous research showing that only a small fraction of a set of audiometric data met the ADBA criteria for analysis (Adera et al, 1993a). How practical is it to use the ADBA procedures for evaluating the HCPs of small- and medium-sized businesses or to subgroups of large populations? The ADBA requirements are so narrow that a large portion of the data collected are rendered unusable in the evaluation. For example, although the study population in the HCP to be evaluated was very large, only 6.8 percent (5576/82,195) of this population and 28.5 percent (5576/19,397) of the population with 3 years of follow-up were eligible to be included in the ADBA method of analysis. Further, a mere 6.2 percent (1193/19,397) of the population with 3 years of follow-up met the entry criteria for cohort-a4, and even fewer (2.7% : 260/9774) of the population with 7 years of follow-up met entry criteria for cohort-a8. This drastically reduced sample size makes it nearly impossible to perform analyses on subgroups of even a large cohort and likely could not be applied to HCPs of most small- and mediumsized businesses. If these figures are reflective of the US worker population, we estimate that an HCP must have a minimum of 485 workers (i.e., [30 x 19,397]/1193) with 3 years of follow-up or 1100 workers (i.e., [30 x 9774]/260) with 7 years of follow-up to be able to use the ADBA method. Such a strict data requirement limits evaluation of the HCPs of many small- and medium-sized businesses, the total work force of which is below 2000 (i.e., 30/[1193/82,195]). This information may be helpful to small- and medium-sized organizations for deciding the adequacy of their sample to application of the ADBA procedures. Generalizability Are the results of the ADBA procedures generalizable to the entire study population? The answer to this question would be "yes" only if the sample chosen, using the ADBA selection criteria, is representative of the study population. We do know from the selection process that the sample was not selected at random. A related issue is whether the risk of developing hearing loss in those who stayed long enough in the program to qualify for the restricted ADBA criteria was similar to those who did not qualify. We reported in a previous study (Adera et al, 1993a) that the relative risk of hearing loss for those who did not meet the ADBA criteria was 9.1 as opposed to a relative risk of 2.3 for those who met the criteria and concluded that the ADBA procedures may systematically exclude employees at high risk of hearing loss from analyses. There appears to be a systematic selection bias, which may render the results nongeneralizable and may lead to an erroneous assessment of an HCP Thus, generalizability of these results to the entire study population appears to be inappropriate when so few nonrandomly selected individuals contribute to the evaluation. Reliability Results of the ADBA analyses showed that the HCP ratings ran the gamut from "unacceptable" to "acceptable" (scale : unacceptable, marginal, acceptable) from year to year and from procedure to procedure. For example, the standard deviation procedure shown in Table 5 reveals that 22 percent, 22 percent, and 56 percent of the test comparisons were rated as "unacceptable," "marginal," and "acceptable," respectively. In addition, we observed that the %Ws and the %BWs procedures generally assigned a lower rating to the program than either of the standard deviation procedures. Low percentage of agreements were observed when the %Ws procedure was compared to the STDVs (kw = 25%) and the STDVAs (kw = 15%) procedures. These low levels of agreement observed between the ADBA procedures reflect poor validity. In conclusion, acceptance of the ADBA method as a national standard may be unwarranted because results from this method (1) yield inconsistent results, (2) may not be

9 Journal of the American Academy of Audiology/Volume 6, Number 4, July 1995 generalizable to the entire study population, and (3) appear impractical for use by mediumand small-sized organizations because of drastically reduced sample size. Instead, we recommend an alternative technique, which has been in use in the field of epidemiology for a long time and is virtually free of the shortcomings of the ADBA method (Kelsey et al, 1986 ; Hennekens and Buring, 1987 ; Kahn and Sempos, 1989). Recently, we presented this method as a way of evaluating HCPs (Adera et al, 1993b). The method utilizes a comparable nonindustrial noise-exposed population as a reference and provides the excess risk of developing hearing loss in the industrial noise-exposed population relative to the reference population. Acknowledgment. This study was supported in part by a grant from the US Army Environmental Hygiene Agency (USAEHA). Grateful appreciation is also due to LTC M. Pengelly and Dr. Doug Ohlin of the Bio-Acoustics Division of the USAEHA. In addition, Mr. Dwayne Oland, Ms. Leslie Eyler, and Ms. Julie Shadoan of the Occupational Health Management Information System at the Ft. Detrick Directorate of Information Management have provided database support. The opinions and assertions contained herein are the private ones of the authors and are not to be construed as official or reflecting the views of the Uniformed Services University of the Health Sciences or the US Army Environmental Hygiene Agency. REFERENCES Adera T, Donahue AM, Malit BD, Gaydos JC. (1993a). Assessment of the proposed Draft American National Standard method for evaluating the effectiveness of hearing conservation programs. J Occup Med 35 : Adera T, Donahue AM, Malit BD, Gaydos JC. (1993b). An epidemiologic method for assessing the effectiveness of hearing conservation programs using audiometric data. Milit Med 158: American National Standards Institute. (1983). Specification of Sound Level Meters. (ANSI ). New York: ANSI. American National Standards Institute, Accredited Standards Committee S12, Noise. (1991). Draft American National Standard Evaluating the Effectiveness of Hearing Conservation Programs. (Draft ANSI ). New York : Acoustical Society of America Through the American Institute of Physics. Department of the Army, Medical Services. (1991). Hearing Conservation. (Pamphlet ). Washington, DC : Headquarters, Department of the Army. Fleiss JL. (1981). Statistical Methods for Rates and Proportions. New York : Wiley. Hearing Evaluation Automated Registry System. (1991). Audiometer Operation Manual : User's Guide. Aberdeen Proving Ground, MD : United States Army Environmental Hygiene Agency. Hennekens CH, Buring JE. (1987). Epidemiology in Medicine. Boston : Little, Brown and Company. Kahn HA, Sempos CT. (1989). Statistical Methods in Epidemiology. New York : Oxford University Press. Kelsey JL, Thompson WD, Evans AS. (1986). Methods in Observational Epidemiology. New York : Oxford University Press. Melnick W. (1984). Evaluation of industrial hearing conservation programs : a review and analysis. Am Ind Hyg Assoc J 45 : Occupational Safety and Health Administration. (1983). Occupational noise exposure ; hearing conservation amendment, final rule, Fed Reg 48 (March 8), Royster LH, Lilley DT, Thomas WG. (1980). Recommended criteria for evaluating the effectiveness of hearing conservation programs. Am Ind Hyg Assoc J 41 : Royster JD, Royster LH. (1986a). Using audiometric data base analysis. J Occup Med 28 : Royster JD, Royster LH. (1986b). Education and motivation. In: Berger EH, ed. Noise and Hearing Conservation Manual, 4th ed. Akron, Ohio : American Industrial Hygiene Association. Royster JD, Royster LH. (1988). Getting started in audiometric data base analysis. Semin Hear 9: Royster JD, Royster LH. (1990a). Hearing conservation programs. In : Harris CM, ed. Handbook of Acoustical Measurements and Noise Control, 3rd ed. New York : McGraw-Hill. Royster JD, Royster LH. (1990b). Important elements and characteristics of hearing conservation programs and determination of their effectiveness. Environ Int 16 : Royster JD, Royster LH. (1990c). Hearing Conservation Programs : Practical Guidelines for Success. Chelsea, MI : Lewis Publishers. Royster JD, Royster LH. (1991). New draft ANSI standard enhances efforts in hearing conservation. Occup Health Saf 60(10):86, Simpson TH, Stewart M, Kaltenbach JA. (1993). Effects of audiometric threshold step size on proposed ANSI outcomes for characterizing hearing conservation program effectiveness. J Am Acad Audiol 4: Suter AH. (1989). The need and benefits of audiometric data base analysis. Sound and Vibration December: