Variance -Estimates and Confidence Intervals for the Kappa Measure of Classification Accuracy

Transcription

1 This file was ceated by scanning the pinted publication. Eos identified by the softwae have been coected; howeve, some eos may emain. Vaiance -Estimates and Confidence Intevals fo the Kappa Measue of Classification Accuacy by M.A. Kalkhan R.M. Reich R.L. Czaplewski RESUME On utilise souvent /'analyse statistique Kappa pou caacteise les esultats d'une evaluation d'exactitude destinee a evalue les classifications d'occupation des sols et de couvetue teeste obtenues a pati de donnees de teledetection. Cette analyse statistique pemet de.compae diffeentes conceptions d'echantillons, les algoithmes de classification, Jes photointepetateus, etc. Pou pouvoi faie ces compaaisons, ii est impotant de savoi quelle peut ete l'etendue de /'eeu d'estimation. On y aive en constuisant des intevalles de confiance autou des points d'estimation. La decision d'utilise des fomules de vaiance asymptotique ou des estimations de vaiance d'amo<;age pou constuie des intevalles de confiance pou /'analyse statistique Kappa n'est pas facile. Celle elude avait pou objet d'aide a touve une eponse a cette question. Neu/ matices d'eeu epesentant tois deges d'exactitude (mauvaise, moyenne et bonne) des donnees TM du satellite Landsat consistant en 4, 8 et 16 categoies de couvetue teeste en Caoline du Nod ant ete utilisees dans cette etude. Chaque matice d'eeu a ete echantillonnee, avec emplacement, a /'aide d'echantillon de 50, 100, 150, 300 et 800 pixels pou obteni /es estimations de /'analyse statistique Kappa et les estimations de vaiance d'amo;age. Chaque matice d'eeu d'echantillon a ete e-echantillonnee 500 fois pou obteni des estimations de vaiance d'amo;age. Lafomule de vaiance asymptotique pou /'analyse statistique Kappa et la vaiance d'amo;age ont founi des estimations sans biais de la vaiance de l'echantillon. En geneal, Jes estimations de vaiance asymptotique etaient plus gandes que celles obtenues 4 ]'aide de l'amo;age, mais la diffeence etait peu impotante. Les intevalles de confiance bases su Jes centiles de la distibution d'amo;age ontfouni les meilleus 95 pou cent de taux de couvetue (92 a 96 % avec une mediane de 95 %). Les 95 % /es plus bas de taux de couvetue ont ete obtenu a /'aide de /'estimation de vaiance d'amo;age (mediane de 83 %). SUMMARY The Kappa statistic is fequently used to chaacteize the esults of an accuacy assessment used to evaluate land use and land cove classifications obtained by emotely sensed data. This statistic allows compaisons of altenative sampling designs, classification algoithms, photointeyetes, and so foth. In ode to make these compaisons, it is impotant to know how fa in eo the estimate might easonably be. This is accomplished by constucting confidence intevals aound the point estimate. The decision to use eithe the asymptotic vaiance fomulae o bootstapping vaiance estimates in constucting confidence intevals fo the Kappa statistic is not a simple task. This study was designed to help answe this question. Nine eo matices epesenting thee levels of accuacy (poo, aveage, pnd good) of Landsat TM Data consisting of 4, 8 and 16 land cove types in Noth Caolina wee used in this study. Each eo matix was sampled, with eplacement, using sample sizes of 50, 100, 150, 300 and 800 pixels to obtain estimates of the Kappa statistic and sample vaiance. Each of the sample eo matices wee esampled 500 times to obtain bootstap estimates of the vaiance. The asjmptotic vaiance fomula fo the Kappa statistic and bootstap vaiance povided unbiased estimates of the sample vaiance. In geneal, the asymptotic vaiance estimates wee lage than those obtained using bootstapping, even though the diffeences wee not significant. Confidence intevals based on pecentiles of the bootstap distibution povided the best 95 pecent coveage ates (92 to 96 pecent with a median of 95 pecent). The lowest 95 pecent coveage ates wee obtained using the bootstap vaiance estimate (median of 83 pecent). M.A. Kalkhan, Ph.D is with the Natual Resouce Ecolo Laboatoy, Coloado State Univesity, Fot Collins, CO 80523, USA Robin M. Reich, Ph.Dis with the Depatment of Foest Sciences Coloado State Univesity,Fot Collins, CO 80523, USA; Raymond L. Ph.D Czaplewski is with the USDA Foest Sevice Rocky Mountain Foest and Range Expeiment Station, 240 Pospect Road, Fot Collins, CO 80526, USA Canadian Jounal of Remote Sensing/Jounal canadien de teledetection

2 INTRODUCTION The most common index used to assess the accuacy of emotely sensed data is the Kappa statistic. The Kappa statistic, which was oiginally developed to measue obseve ageement fo categoical data (Cohen 1960) has eceived consideable attention in emote sensing applications (Congalton and Mead 1983; Rosenfield and Fitzpatick-Lins 1986; Aickin 1990; Congalton 1991; Stehman 1992; Fitzgeald and Lees 1994; and Kalkhan 1994). The Kappa statistic is defined as (Bishop et al. 1975, p ): NL xii - L x;.x.; K = 1=1 1=1 N2 - I X;+X+i 1=1 whee is the numbe of ows in the eo matix x.. is the ' II numbe of obsevations in ow i and column i (i.e. the diagonal elements), X;+ and x.; ae maginal totals ofow i and column i 1 espectively, and N is the total numbe of obsevations. If thee is pefect ageement between categoies, K = 1, while a value of K = 0 indicates that the obseved ageement equals chance ageement (Cohen 1960). Skidmoe and- Tune (1989) point out that positive values of Kappa_;c~u fom geate than chance ageement, while negative V!:Ilues indicates a less than chance ageement. The lowe limit of the Kappa statistic depends on the maginal distibutions and is not likely to have pactical inteest (Rosenfield and Fitzpatick-Lins 1986). Using citeia developed by Landis and Kock (1977), Monseud and Leemans (1992) suggest that a value of Kappa geate than,0.75 indicates vey good to excellent ageement, while a value between 0.4 and 0.75 indicates fai to good ageement. A value less than o equal to 0.4 indicates poo ageement between classification categoies. The asymptotic vaiance of the Kappa statistic is given by,(bishop et al. 1975, p.396): I 'whee, I I 03 = L Pu(P1+ + P +1), I=! p.,=:.jl I N' 02 =I P1+ p +i l=l (1) (2) 04 =I I P/P1+ + p +1) 2 l=lj=l p =Xi+ t+ N' p = X+I +t N' p = X!f+!I N' In applying the Kappa statistic, it is assumed that one is sampling fom a.multinomial distibution in which each sampling unit is classified into a mutually exclusive categoy (Bishop et al. 1975; Congalton 1991; Stehman 1992; Kalkhan 1994). With a lage sample size, the nomal appoximation holds easonably well in tems of constucting confidence intevals and hypothesis testing. With small sample sizes this may not be tue. To ovecome this potential poblem associated with small sample sizes, one can use bootstapping (Efon 1979) to obtain unbiased estimates of the sample vaiance. This is accomplished by geneating B bootstap eo matices, each consisting of N data values dawn with eplacement fom an eo matix consisting of N objects. Fo each bootstap eplication of the eo matix, the Kappa statistic is computed using Equation 1. The vaiance of the Kappa statistic is obtained using the empiical vaiance of the B bootstap eplication (Efon and Tibshiani 1993). In geneal, bootstap vaiance estimates may povide bette estimates of the vaiance than those obtained fom using the asymptotic vaiance fomula (Equation 2). The estimated vaiances ae often used to assign appoximate confidence intevals to a paamete of inteest. If the distibution of the Kappa statistic is non-nomal, the use of the standad nomal distibution in constucting confidence intevals may not adjust the confidence inteval to account fo skewness in the undelying distibution, o othe eos that can esult when estimating the Kappa statistic (Efon and Tibshiani 1993). As an altenative, one could use the pecentiles of the bootstap histogam to define confidence intevals without having to make nomal theoy assumption. If the bootstap distibution of the Kappa statistic is oughly nomal, then the standad nomal and pecentile inteval will geneally agee. If the bootstap distibution is non-nomal, then the pecentile inteval should achieve a bette balance in the left and ight tails, since one is using moe of the infomation in the bootstap histogam than just its standad deviation (Efon and Tibshiani 1993). Pevious studies aimed at evaluating the statistical popeties of the Kappa statistic in assessing the accuacy of emotely sensed imagey have focused pimaily on the bias of the statistic (Stehman 1992). Little o no attention has been given to the statistical popeties of the vaiance estimato and accompanying confidence inteval. Thus, the objective of this study is to evaluate the use of bootstapping to obtain confidence intevals and vaiance estimates of the Kappa statistic in assessing the accuacy of emotely sensed data. METHODS Study Aea The study aea is located in the State of Noth Caolina. Noth Caolina was selected because of its divesity in physiogaphic egions, epesenting land cove conditions commonly found in the easten and southen United States. Elevations ange fom sea level to ove 2,000 metes. Kuchle (1985) stated that potential climax vegetation fo most of the entie state is

3 t J ~MMMtiiM#MMti!M@E-i Appalachian oak foest. Mountainous aeas include foest species commonly found in moe nothen latitudes. The flat coastal plain includes sand idges, bays pocosins, and maitime foests. Relative to othe tempeate foests, vegetation in Noth Caolina is vey divese. Czaplewski et al. (1987) povides a moe complete desciption of the study aea. Data The data used in this study wee fom a pilot study designed to evaluate the use of lage pemanent sample plots and Landsat TM data to monito shot-tem land cove and land use changes at the state and egional level (Scheude et al. 1986). Sample plots wee located on a systematic gid whee each node of the gid coesponded to the appoximate cente of altenate ows of a 7.5 minute USGS topogaphic quadangle, fo a total 411 pemanent sample plots each with an aea of 405 ha ( 4500 pixels). Sample plots wee classified with espect to thei land cove and land use using basic land cove categoies (Level I) adopted by the Common Teminology Wok Goup (Powell 1981 ). Foest categoies which epesent Level II classifications wee developed in association with the Southen Foest Inventoy Analysis Unit to ensue the availability of petinent foest wide infomation. Image Analysis and Classification A Landsat TM scene (Octobe 8, 1985, Identification numbe Y XO) nea Raleigh, Noth Caolina was selected fo this study. The scene contained 35 sample plots. Sixteen cove types wee identified using an unsupevised classification pocedue available in ERDAS Softwae (vesion 7.5, 1992) using bands TM3 (ed), TM4 (IRl), and TM5 (IR2) (Table 1). Of the 35 plots, fou wee selected to epesent images with a poo classification and fou wee selected to epesent an aveage classification. Citeia used in selecting the sample plots wee based on Came's V (Bishop et al. 1985, p. 386) coefficient of ageement. Plots selected to epesent an image with a poo classification had a Came's V anging fom to 0.178, while the plots epesenting an aveage classification had a Came's V anging fom to The intepetation of Came's V is simila to that of the Kappa statistic, but Came's V is not as widely used in the emote sensing liteatue. The sample plots epesenting the poo and aveage classifications wee combined to fom a 16x16 composite eo matix epesenting the oveall accuacy of each goup of plots. Each eo matix consisted of a seies of ows and columns epesenting the cove types identified on emotely sensed imagey (columns) and the gound (ows). The eo matix povides the uses with infomation on the accuacy of identifying individual cove types and both eos of commission and omission in the classification (Rosenfield and Fitzpatick Lins 1986). Eos of commission elate to the accuacy of the aeial photogaphs while eos of omissi9n epesent the accuacy of the emotely sensed data. The eo matices wee then collapsed to fom a set of 8x8 and 4x4 eo matices. The collapsed eo matices wee ceated by combining classes with simila class signatues (Table 1). Because of the low accuacy associated with the sample data, we developed a thid set of eo matices epesenting a good level of ageement in the cove type maps. This was accomplished by building up the diagonal of the individual eo matices. In building up the diagonals we educed the cell counts of the off-diagonal cells associated with obvious misclassification eos such as gassland and pine sawtimbe, while cell counts associated with Table 1. Initial and educed classification scheme used to descibe the land cove and foest types in Noth Caolina. Initial 16 classes Reduced 8 classes 1. Idle agicultue, cop land, and Idle agicultue, cop land, nonstocked foest land nonstocked foest and seedlings/saplings (1,13,14,15,16) 2. Gassland 3. Uban Gassland (2) 4. Pine sawtimbe Uban (3) 5. Mixed sawtimbe Pine sawtimbe (4). 6. Pine sawtimbe Mixed sawtimbe ( 5) 7. Bottom land sawtimbe. Oak-pine sawtimbe (6) 8. Upland sawtimbe Hadwood sawtimbe (7,8) 9. Pine and mixed pole timbe Pole timbe (9,10,11,12) 10. Oak-pine pole timbe 11. Bottom land pole timbe 12. Doland pole timbe 13. Pine and mixed seedlini:dsaolings 14. Oak seedlin!usaolimzs 15. Bottom land seedlings/saolings 16. Upland seedlings/saplings Reduced 4 classes Nonfoested, seedling/saplings (1, 2, 13, 14, 15, 16) Uban (3) Pine and mixed sawtimbe and pole timbe (4,5,9) Oak-hadwood sawtimbe and pole timbe (6,7,8,10,l l,12)

4 less obvious misclassification eos such as pine sawtimbe and pine pole timbe wee left unchanged. Finally, all eo matices wee conveted to joint pobability matices fo the pupose of sampling. This was accomplished by dividing the individual cell :ounts by the total numbe of sample plots.. Data Analysis Each of the nine composite eo matices ( 3 levels of accuacy X J levels of classes) wee sampled n = 50, 100, 150, 300, and ~00 times using a two-step pocess to appoximate an equal Jobability sample. Fist, a cove type on the emote sensing mage was selected with pobability popotional to thei naginal pobabilities (columns). The coesponding gound ;lassification was then selected with pobability popotional to he conditional pobability of obseving a paticula cove type ~iven the emote sensing classification obtained in the fist ;tep. This was epeated n times to obtain an estimate of the mo matix which was then used to obtain estimates of the K.appa statistic (Equation 1), its vaiance (Equation 2) and 95% ;onfidence inteval assuming a nomal distibution. This Jocess was epated 500 times fo a total of 22,500 estimated mo matices. Each of the estimated 22,500 eo matices wee esampled B =. 500 times using a sample of size n to obtain 500 estimates >fthe Kappa statistic. The vaiance of the 500 estimates of the K.appa statistic became the bootstap vaiance estimate of the Kappa statistic. The bootstap vaiance estimates wee used to ;onstuct 95% confidence intevals aound the estimated Kappa statistic, assuming a nomal distibution. Bootstap Jecentile intevals (0.025, 0.975) wee also computed fom the Jootstap histogam of the Kappa statistics. The bias fo the Kappa statistic was computed as the diffeence between the aveage of the 500 estimates of the Kappa statistic l.!;ild thei tue values. A t-test was used to test the null hypothesis of no significant bias at the 0.05 level of significance. In testing this hypothesis the vaiance of the 500 estimates of the Kappa statistic (i.e. simulation vaiance) was used as an estimate of the sample vaiance. In addition to knowing whethe an estimate is unbiased, it is desiable to know something about the distibution of the sample statistic. The Shapio and Wilk's (1965) Wtest was used to test the null hypothesis that the Kappa statistic is nomally distibuted at the 0.05 level of significance. Randomly selected bootstap distibutions wee also tested fo nomality. The selected bootstap distibutions wee compaed to the sample distibution of the Kappa statistic using the two sample Kolmogoov-Smimov goodness-of-fit statistic. The bias associated with estimating the sample vaiance was evaluated by computing the atio of the mean vaiance to the simulation vaiance. The mean vaiance of the asymptotic vaiance (Equation 2) and bootstap vaiance wee computed by aveaging the 500 estimated vaiances. The atio of the asymptotic vaiance to the bootstap vaiance was also computed. An F-test was used to test fo significant diffeences in the vaiance estimates at the 0.05 level of significance. Finally, we detemined the popotion of confidence intevals computed using the classical vaiance, bootstap vaiance, and bootstap pecentile intevals that enclosed the tue Kappa. RESULTS AND DISCUSSION The nine eo matices had Kappa statistics anging fom a low of0.35 pecent to a high of82.06 pecent (Table 2). In geneal, 4 8 Table2. Influence of sample size, accuacy of the emotely sensed image, and numbe of cove types in estimating the Kappa. statistic. The numbes in paenthases ae the standad deviations associated with individual estimates of Kann a. Estimates of the Kanna statistic and standad deviations wee based on 500 Monte Calo simulations. Aveage Estimated Kappa(%) (standad deviation(%)) No. Level of Tue Classes Ageement Kanoa :.(~l m ~(} M~!ml." 150,..1J!Q " ---~--80~0 Poo (9.53) 0.20* (6.77) 0.34 (5.08) 0.16* (3.66) 0.41 * (2.30) Aveage 3571% 35.38* (9.28) 35.03* (6.39) 35.23* (5.29) (3.72) 35.76* (2.23) Good * (6.90) (4.68) (3.83) 82.25* (2.78) 81.97* (1.60) Poo * (7.69) 5.19 (5.34) 5.17 (4.10) 5.35* (2.42) 5.18 (1.83) Aveage * (8.00) 30.65* (5.48) * (4.40) (3.08) (2.08) Good (6.76) (4.67) 77.70* (3.85) 77.89* (2.77) 77.75* (1.57) 16 Poo (4.24) 1.53 (2.47) 1.39* (2.38) 1.42* (1.82) 1.51 * (1.02) Aveage * (6.18) 20.53* (4.45) 20.15* (3.56) * (2.73) 20.27* (1.57) Good * (5.99) 74.43* (5.03) 74.09* (3.98) 74.35* (2.83) 74.10* (1.68) * The estimated Kappa diffeed significantly fom the tue Kappa based on a two-tailed t-test with 499 degees of feedom at the 0.05 level.

5 the sample estimates of the Kappa statistic wee biased, though thee was no consistency in the diection of this bias. Thee was also no elationship between the sample size and numbe of classes and whethe an estimate was biased o not. Thee was howeve, a tendency fo the eo matices epesenting a emote sensing image with an aveage and good level of accuacy to have a highe popotion of biased estimates than the eo matices epesenting a poo level of accuacy (Table 2). While these biases niay not seem lage fom a pactical point of view, they can have a diect impact on the obseved coveage ates by distoting the confidence pobabilities (Cochan 1977, p ). In a simila study, Stehman (1992) obseved little to no bias associated with the Kappa statistic in estimating the accuacy ofemote sensing images. Howeve, Stehman (1992) only consideed eo matices with thee classes and Kappa statistics anging fom 45 to 68 pecent. Because of the diffeences in the numbe of classes and the anges in the level of accuacy, the esults of these two studies may not be diectly compaable. The asymptotic vaiance fomulae (Equation 2), fo the most pat, povided unbiased estimates of the sample vaiance (Table 3). Exceptions to this occued when the level of accuacy associat~d with the population eo matix had a Kappa statistic less than 1.6 pecent. In these instances, the asymptotic vaiance fomulae significantly oveestimated the sample vaiance, iespective of the sample size o numbe of classes associated with the eo matix. Tests of nomality indicated that 38 out of 45 sample distibutions of the Kappa statistic wee nomally distibuted (Table 3). In the few cases in which the hypothesis of nomality was ejected, the sample distibutions wee skewed to the ight, and the asymptotic vaiance fomulae tended to slightly undeestimate the sample vaiance. The bootstap vaiance estimate, in contast, povided unbiased estimates of the sample vaiance, iespective of the sample size, level of accuacy, o the numbe of classes associated with the population eo matix (Table 4). Thee was a geneal tendency of the bootstap vaiance estimate to undeestimate the sample vaiance. A few of the bootstap distibutions did not follow a nomal distibution (Table 4). In these cases,. the bootstap distibutions wee skewed to the left,justthe opposite of what was obseved fo the sample distibutions of the Kappa statistic. The asymptotic vaiance estimates wee significantly lage than the bootstap vaiance estimate fo population eo matices with a Kappa statistic less than 1.6 pecent. In geneal though, the asymptotic vaiance estimates tended to be lage than those obtained using bootstapping, even though these diffeences wee not significant. This suggests that the bootstap distibutions have less vaiability associated with them compaed to the oiginal eo matices. When we compaed the cumulative density function of the sample estimates of the Kappa statistic with those obtained fom the bootstap pocedue, 41 out of 45 wee significantly diffeent at the 0.05 level. Compaing the individual distibutions, we noticed that the sample distibutions wee slightly skewed to Table3. Ratio of the asymptotic vaiance to the simulation vaiance of the Kappa statistic. Table 4. Ratio of the mean bootstap vaiance to the simulation. vaiance of the Kappa statistic. No. Level of I Classes Ageement 50 I 'too '800 No. Level of Classes Ageement Poo 1.26* * Aveage Good 0.93# 1.00# Poo Aveage # Good 0.95# 1.00# Poo 1.58*# 1.61 * * 1.52 Aveage Good # * The asymptotic vaiance diffeed significantly fom the simulation vaiance using a two-tailed F-test with 499 degees of feedom in the numeato and denominato at the 0.05 level. #The sample distibution of the Kappa statistic diffeed significantly fom a nomal distibution using the Shapio-Wilk's (1965) W test at the 0.05 level. 4 Poo Aveage Good # Poo Aveage Good Poo 0.92# 1.05# Aveage 0.98# Good * The bootstap vaiance diffeed significantly fom the simulation vaiance using a two-tailed F-test with 499 degees of feedom in the numeato and denominato at the 0.05 level. # The bootstap distibution of the Kappa statistic diffeed significantly fom a nomal distibution using the Shapio-Wilk's (1965) W test at the 0.05 level.

6 ~., ~" M~ ft!til! llm'~" "IM&~.! l~l~! ii~"~m~fi.:mnnm~ iliilllillill~fi.1.:ii~h~.l ' '' " IM!il~ tj ij~ ~M lliiil.. Table 5. Ninety -five pecent confidence coveage ates fo the Kappa statistic using asymptotic vaiance unde the assumption of nomality. No. Level of Classes Ageement Fequency 12 A Poo Aveage Good Poo Aveage Good Poo Aveage Good B 2 Fequency Coveage Rate (%) the ight, while the bootstap distibutions wee skewed somewhat to the left. Also, the bootstap distibutions wee flatte than the sample distibutions which may account fo the slightly smalle vaiance estimates. Ninety-five pecent confidence coveage ates fo the estimated Kappa statistic using the asymptotic vaiance fomulae anged fom 88 to 96 pecent (Table 6). The median coveage ate was 95 pecent. Figue la depicts the fequency distibution of the coveage ates obtained using the asymptotic vaiance. Eo matices with a low level of accuacy geneally had coveage vites lage than expected, while eo matices with a high level Fequency 25 c Coveage Rate(%) Table 6. Ninety-five pecent confidence coveage ates fo the Kappa statistic using bootstap vaiance estimates unde the assumption of nomality. No. Level of t Classes Ageement i ,. 4 Poo Aveage Good Poo Aveage ' Good Poo Aveage Good Coveage Rate (%} Figue 1. Fequency distibution of the 95 pecent coveage ates obtained using (A) the asymptotic vaiance fomulae, (B) the bootstap vaiance estimate, and (C) the pecentiles of the bootstap distibution. of accuacy had coveage ates lowe than expected. The lage than expected coveage ates obseved fo eo matices with a low level of accuacy was diectly attibutable to the ove estimation of the sample vaiance (Table 3). Similaly, the lowe than expected coveage ates associated with eo matices with a high level of accuacy was due to the tendency of the asymptotic vaiance fomulae to undeestimate the sample vaiance and the bias associated with the sample statistic. Both of these factos can esult in a distotion in the obseved confidence pobabilities. In contast, the coveage ates using th.e bootstap vaiance estimates anged fom 80 to 87 pecent, with a median coveage ate of 83 pecent (Table 6, Figue lb). The lowe than 11111

7 expected coveage ates may be due in pat to the geneal tendency of the bootstapping pocedue to undeestimate the sample vaiance (Table 4). Thee is some anecdotal evidence to suggest that the undeestimation of the sample vaiance may be due to the sampling pocedue used in obtaining the bootstap sample (Scheude 1996, pesonal communications). Altenative sampling pocedues may impove the vaiance estimates. The biases associated with estimating the Kappa statistic also may be a contibuting facto to the lowe than expected coveage ates. Confidence intevals based on the pecentiles of the bootstap distibution povided the best coveage ates (92 to 96 pecent with a median of 95 pecent) of the thee pocedues evaluated in this study (Table 7, Figue lc). The pecentiles of the bootstap distibution obtained a bette balance in the left and ight tails since they use moe infomation than just an estimate of the vaiability associated with the distibution. Table 7. Ninety-five pecent confidence coveage ates fo the Kappa statistic based on the pecentiles of the bootstap distibution. No. Level of t.. Classes Ageement 50 Joo Poo Aveage Good Poo Aveage Good Poo Aveage Good RECOMMENDATIONS Based on the esults of this study, we ecommend the following fo estimating the sample vaiance and constucting confidence intevals fo the Kappa statistic: 1. Use the bootstap vaiance estimate to obtain unbiased estimates of the sample vaiance fo puposes of hypothesis testing. We do not ecommend using the bootstap vaiance estimates fo constucting confidence intevals. 2. Use the asymptotic vaiance fomulae fo constucting confidence intevals assuming a nomal distibution. 3. Pecentiles of the bootstap distibution should be used in constucting confidence intevals fo the estimated Kappa statistic. REFERENCES Aicldn, M "Maximum likelihood estimation of ageement in the constant pedictive pobability model, and its elation to Cohen's Kappa", Biometics, 46: ( Bishop, Y. M. M., S. E. Feinbeg, and P. W. Hooland "Discete multivaiate analysis theoy and pactice", MIT Pess, Cambidge, MA, 575 p. Cochan, W. G "Sampling techniques'', 3d ed. John Wiley and Sons, New Yok, 428 p. Cohen, J "A coefficient of ageement of nominal scales", Education Psychological Measuements, 20: Congalton, R. G., and R. A. Mead "A quantitative method to test fo consistency and coectness in photointepetation", Photogammtic Engineeing and Remote Sensing, 49: Congalton, R. G "A eview of assessing the accuacy of classification of emotely sensed data" Remote Sensing of Envionment, 37: Czaplewski, R. L., G. L. Catts, and P. W. Snook "National land cove monitoing using lage, pemanent photo plots", In: Land and Resouce Evaluation fo National Planning in the Topics. An Intenational Confeence and Wokshop, Chetumal Mexico, Januay 25-31, 1987, 524 p. Efon, B "Bootstap methods: Anothe look at the jackknife", Annals of Statistic, 7:1-26. ' Efon, B. And R. J. Tibshiani "An intoduction to the bootstap", Chapman & Hall, 431 p. ERDAS, Inc Bufod Highway, Suite 300, Atlanat, Geogia 30329, USA. Fitzgeald, R. W. and B. G. Lees "Assessing the classification accuacy of multisouce emote sensing data", Remote Sensing of Envioment, 47: Kalkhan, M. A "Statistical popeties of six accuacy indices using simple andom and statified andom sampling: An application in emote sensing", Ph.D. Dissetation. Coloado State Univesity, 134 p. Kuchle, A. W "Potential natual vegetation", National Atlas of the United States. U.S. Depatment of Inteio, U.S. Geological Suvey, (1 map sheet). Landis, J. R., and G. G. Kock "The measuement of obseve ageement fo categoical data", Biometics, 33: Monseud, R. A. and R. Leemans "Compaing global vegetation' maps with the Kappa statistic", Ecological Modeling, 62: Powell, D. S "A solution fo s~lving conflicts between land use and land cove inventoies", P In: Place Resouce Inventoies, Pinciples and Pactices, National SAF Wokshop, August 9-14, Rosenfield, G. H. and K. Fitzpatick Lins "A coefficient of ageement as a measue of thematic classification accuacy", Photogammtic Engineeing and Remote Sensing, 52: Scheude, H. T Rocky Mountain Foest and Range Expeiment Station, USDA Foest Sevice, 240 West Pospect Steet, Fot Collins, CO Scheude, H. T., P. W. Snook, R. L. Czaplewski, and G. P. Catts "A poposed peiodic national inventoy of land-use land cove", National Ameican Society of Photogammety Confeence, Septembe Anchoage, AK. Shapio, S. S. And M. B. Wilk "Analysis of vaiance test fo nomality (complete test)", Biometica, 52: Skidmoe, A., and B. Tune "Assessing the accuacy of esouce inventoy maps", p In: Poc. of Global Natual Resouce Monitoing and Assessment: Pepaing fo the 21st Centuy, Venice, Italy. Stehman, S. V "Compaison of systematic and andom sampling fo;i estimating the accuacy of maps geneated fom emotely sensed data", Photogammtic Engineeing and Remote Sensing, 58: j J