Reliability of store and forward teledermatology for skin neoplasms

Transcription

1 Reliability of store and forward teledermatology for skin neoplasms Erin M. Warshaw, MD, MS, a,b Amy A. Gravely, MA, a and David B. Nelson, PhD a,c Minneapolis, Minnesota Background: Teledermatology may be less optimal for skin neoplasms than for rashes. Objectives: We sought to determine agreement for skin neoplasms. Methods: This was a repeated measures study. Each lesion was examined by a clinic dermatologist and a teledermatologist; both generated a primary diagnosis, up to 2 differential diagnoses, and management. Macro images and polarized light dermoscopy images were obtained; for pigmented lesions only, contact immersion dermoscopy image was obtained. Results: There were 3021 lesions in 2152 patients. Of 1685 biopsied lesions, there were 410 basal cell carcinomas (24%), 240 squamous cell carcinomas (14%), and 41 melanomas (2.4%). Agreement was fair to substantial for primary diagnosis (45.7%-80.1%; kappa ), substantial to almost perfect for aggregated diagnoses (primary plus differential; 78.6%-93.9%; kappa ), and fair for management (66.7%-86.1%; kappa ). Diagnostic agreement rates were higher for pigmented lesions (52.8%-93.9%; kappa ) than nonpigmented lesions (47.7%-87.3%; kappa ), whereas the reverse was found for management agreement (pigmented: 66.7%-79.8%, kappa vs nonpigmented: 72.0%-86.1%, kappa ). Agreement rates using macro images were similar to polarized light dermoscopy; contact immersion dermoscopy, however, significantly improved rates for pigmented lesions. Limitations: We studied a homogeneous population. Conclusions: Diagnostic agreement was moderate to almost perfect whereas management agreement was fair. Polarized light dermoscopy increased rates modestly whereas contact immersion dermoscopy significantly increased rates for pigmented lesions. ( J Am Acad Dermatol 2015;72: ) Key words: dermoscopy; diagnosis; management; reliability; skin cancer; teledermatology. Although several small studies have reported reliability (agreement) of teledermatology with in-person examinations, fewhavefocusedspecificallyonskinneoplasms. A recent systematic review on teledermatology 1,2 found that primary diagnostic agreement has only been assessed in 5 lesion studies 3-7 and 1 dermoscopy pigmented lesion study 8 ; the weighted average for these 6 studies (n = 708 lesions) was 62.3%. 1 The weighted average for aggregated diagnostic agreement for 4 studies 4-7 (n = 358 lesions) was similar, 64.4%. 1 Because almost half From the Minneapolis Veterans Affairs Medical Center, Center for Chronic Disease Outcomes Research a ; and Departments of Dermatology b and Medicine, c University of Minnesota School of Medicine. This research was supported by the Department of Veterans Affairs (VA), Veterans Health Administration, Health Services Research and Development Service IIR During this study, Dr Warshaw was supported by a VA Cooperative Studies Clinical Research Career Development Award. Conflicts of interest: None declared. The findings and conclusions presented in this report are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or Health Services Research and Development Service. Accepted for publication November 3, Reprints not available from the authors. Correspondence to: Erin M. Warshaw, MD, MS, Minneapolis Veterans Affairs Medical Center, Department 111 K, 1 Veterans Dr, Minneapolis, MN erin.warshaw@va. gov. Published online January 16, Published by Elsevier on behalf of the American Academy of Dermatology, Inc

2 JAM ACAD DERMATOL VOLUME 72, NUMBER 3 Warshaw, Gravely, and Nelson 427 (47%) of all veteran dermatology visits 9 are related to skin neoplasms, evaluation of this subset of dermatologic conditions is critical in this patient population. Previous work by our group reported accuracy (histopathology as gold standard) of store and forward teledermatology as compared with conventional in-person dermatologic examinations for skin neoplasms and found that the diagnostic accuracy of teledermatology was inferior to standard, in-person examinations whereas management accuracy varied by lesion type Interobserver accuracy was also reported. 13,14 The purpose of this d analysis was to compare conventional, in-person dermatology with store and forward teledermatology for skin neoplasms, using the outcomes of agreement for primary diagnosis, aggregated diagnoses, and management. CAPSULE SUMMARY METHODS The Minneapolis Veterans Affairs Medical Center Human Studies Subcommittee in Minnesota provided institutional review board approval for this study. The design of the cross-sectional, repeated measures equivalence study has been reported previously Participants and inclusion criteria Neoplasms were defined as circumscribed lesions. Patients referred to the Minneapolis Veterans Affairs Medical Center Dermatology Clinic for evaluation of a skin neoplasm or patients already enrolled in the dermatology clinic who were undergoing a biopsy of a skin neoplasm were eligible for inclusion. After informed consent was obtained, digital photographs of the lesion(s) and a standardized history were obtained by research staff. A staff dermatologist then completed a clinical assessment that consisted of: (1) a choice of 17 common diagnoses for 1 primary diagnosis and up to 2 differential diagnoses; (2) a choice of 4 basic management plans; (3) pigmentation (yes or no); and (4) level of diagnostic confidence (low, moderate, or high). Clinicians were also allowed to choose other for diagnoses and management and hand-write choices. Additional history could be obtained by the clinical dermatologist in the usual manner of a clinic d d Store-forward teledermatology is being implemented but studies focusing on skin neoplasms are lacking. In our series, diagnostic agreement was moderate to almost perfect whereas management agreement was fair. Store-forward teledermatology should be used cautiously in evaluating skin neoplasms; contact immersion teledermoscopy should be used whenever possible for pigmented lesions. encounter and the clinical examination could include all options normally available (eg, palpation, diascopy, dermoscopy). Photographs Research assistants obtained images with up to 3 different cameras. All patients had at least 2 macro images (distance and closeup; digital Nikon Coolpix 4500 with a Nikon SL-1 ring flash [Nikon, Melville, NY]) and 1 polarized light image (polarized light dermoscopy [PLD]) (digital Nikon Coolpix 4500 with a 3Gen Dermlite lens attachment [3Gen, San Juan Capistrano, CA]). For lesions greater than 2 mm in height, an additional macro angle shot was obtained. All of the images were obtained in a standardized fashion (distance pixels; all others at the highest resolution pixels). For pigmented lesions only, a contact immersion dermoscopy (CID) image (35-mm Minolta X 370 [Minolta, Tokyo, Japan] with a Heine dermphot lens attachment [Heine, Herrsching, Germany]) was also obtained. Previous publications have focused on index lesions. 10,11 The study reported herein includes both index and secondary lesions. Image packages For each patient, a sequence of 2 or 3 images, or package, was sent to a teledermatologist according to a computer-generated randomization schedule separated by at least 3 weeks, to avoid recall bias. For patients with only nonpigmented lesions, the teledermatologist received a macro package (distance, close-up, and/or angle [if height [2 mm] image of all lesions) and PLD package (macro and PLD images of all lesions). For patients with pigmented lesions, the teledermatologist received a macro package, a PLD package, and a CID package (macro images of all lesions plus CID images of pigmented lesions). Teledermatology encounter For a given patient, 1 of 3 board-certified dermatologists (none of whom served as general clinic dermatologists but all of whom had clinical expertise in dermoscopy and pigmented lesions) 14 was randomly selected to review the electronically transmitted packages of clinical digital photographs

3 428 Warshaw, Gravely, and Nelson JAM ACAD DERMATOL MARCH 2015 Fig 1. Flow diagram of study participants. and standardized history. The order in which the different packages were sent to the selected dermatologist was randomly selected. Using the same diagnostic and management categories as the clinic dermatologist, the teledermatologist recorded 1 primary diagnosis, up to 2 differential diagnoses, and management. Confidence and image quality was also rated. Panel A separate panel of 3 board-certified dermatologists, none of whom served as a clinic or teledermatologist in the study and who were unaware of the study design and purpose, grouped write-in diagnoses into categories for the purpose of data analysis. Outcome measures Primary diagnostic agreement was defined as exact matching of primary diagnoses only. Aggregated diagnostic agreement was defined as matching of any of the clinic dermatology diagnoses (primary or differential) with any of the teledermatology diagnoses (primary or differential). Management agreement was defined as matching of the teledermatology and clinic dermatology management. Statistical analysis We used standard large sample methods to estimate rates of agreement and kappa statistics (with corresponding 95% confidence intervals) for lesions in the 4 combinations of pigmentation and biopsy for each of the 3 photographic modes considered. Note that we calculated agreement for all lesions for primary, aggregated, and management agreement, and by confidence level and photograph quality. For simplicity and brevity we present these latter results for pigmented and nonpigmented lesions given the generally similar patterns in differences in rates across categories of confidence and image quality for biopsied and nonbiopsied lesions. Comparisons of agreement rates between the categories of pigmentation and biopsy status were made using large sample, normal approximation-based tests (z-tests) ignoring any positive correlation between observations made by the same clinic dermatologist, teledermatologist, or

4 JAM ACAD DERMATOL VOLUME 72, NUMBER 3 Warshaw, Gravely, and Nelson 429 for the same patient. For comparisons of agreement rates for different image types, we used large sample, normal approximation-based tests that incorporated a correlation between observations on the same lesion. Data analysis The study targeted at least 520 individuals with lesions in each category (biopsied pigmented, biopsied nonpigmented, nonbiopsied nonpigmented, and nonbiopsied nonpigmented). This sample size allowed estimation of agreement rates with SE less than 2.5% and 90% power for comparing rates, at a.05 significance level and an underlying 10% difference. Data analysis was performed on a statistical software package (SAS for Windows, SAS Institute Inc, Cary, NC). RESULTS Demographics Participant characteristics. Participant characteristics have been previously reported. 10,11 Briefly, of 2905 volunteers who were invited to participate, 2152 (74%) were eligible and agreed to participate (Fig 1). The majority of participants was male and Caucasian; most reported no personal (58%) or family (73%) history of skin problems (Table I). Of the 259 patients who were approached more than once because they had more than 1 clinic visit and/or consult during the study period, 129 were enrolled twice, 6 were enrolled 3 times, and 1 patient was enrolled 4 times. Of all visits, the majority (1735, 81%) involved only 1 lesion, although 423 (18%) included 2 lesions and 138 (6%) involved 3 or more lesions. Lesion characteristics. Characteristics of lesions are listed in Table II. Overall, almost half (45%) were located on the face or ears. About a quarter of all lesions had been present for 3 to 12 months (24%). Approximately two fifths (43%) of lesions were asymptomatic whereas one-quarter had changed in size (25%) or were pruritic (24%). Histopathologic diagnostic categories for biopsied lesions are summarized in Table III. Approximately one fourth of biopsied lesions were basal cell carcinomas (24%) followed by squamous cell carcinomas (14%) and benign keratoses (13%). There were 41 melanomas. Primary diagnostic agreement For standard macro images, diagnostic agreement for primary diagnosis between teledermatologists and clinic dermatologists ranged across the lesion categories from 45.7% to 75.7% (Table IV). Agreement rates were significantly higher for nonbiopsied pigmented lesions than for the other 3 Table I. Demographic characteristics of enrolled patients Unique enrolled Characteristic patients, n = 2152 Male gender, n (%) 2082 (96.8) Mean age, y (range) 68 (19-94) Fitzpatrick skin type, n (%)* I 209 (9.7) II 609 (28.3) III 1025 (47.6) IV-VI 308 (14.3) Ethnicity, n (%) y Caucasian 2098 (97.5) African American 32 (1.5) American Indian 8 (0.4) Hispanic/Latino 8 (0.4) Asian 2 (0.1) Other 2 (0.1) Personal history, n (%) zk Nonmelanoma skin cancer 567 (24.9) Melanoma 77 (3.4) Eczema 91 (4.0) Psoriasis 75 (3.3) Other skin problem 150 (6.6) None 1321 (57.9) Family history, n (%) xk Nonmelanoma skin cancer 295 (13.2) Melanoma 119 (5.3) Eczema 81 (3.6) Psoriasis 70 (3.1) Other skin problem 39 (1.7) None 1637 (73.1) *Not specified in 1 patient. y Not specified in 2 patients. z Not specified in 4 patients. x Not specified in 2 patients. k n = Number of patients who answered yes to each question; therefore the totals in this section exceed the number of patients. categories (P \.0001) and significantly lower for biopsied nonpigmented lesions (P \.025). Rates of agreement with the addition of the PLD images were comparable with those with just the macro images, except for biopsied nonpigmented lesions where the rate was significantly higher (P =.0046). The addition of CID images significantly increased the rates of agreement for pigmented lesions (all P values \.022). Kappa statistics were fair to moderate. Aggregated diagnostic agreement For the standard macro images, aggregated diagnostic agreement ranged between 78.6% and 91.0% with significantly lower rates (P values\.001) of agreement for nonbiopsied nonpigmented lesions and significantly higher rates (P values \.001) of

5 430 Warshaw, Gravely, and Nelson JAM ACAD DERMATOL MARCH 2015 Table II. Lesion characteristics Biopsied, n = 1685 Nonbiopsied, n = 1336 Characteristic Pigmented, n = 651 Nonpigmented, n = 1034 Pigmented, n = 753 Nonpigmented, n = 583 Total, n = 3021 Location (% of answers in column) Face/ears 164 (25.2) 600 (58.0) 269 (35.7) 315 (54.0) 1348 (44.6) Back 210 (32.3) 85 (8.2) 184 (24.4) 25 (4.3) 504 (16.7) Hand/arm 66 (10.1) 141 (13.6) 76 (10.1) 105 (18.0) 388 (12.8) Chest 66 (10.1) 85 (8.2) 64 (8.5) 17 (3.0) 203 (6.7) Scalp 38 (5.8) 57 (5.5) 47 (6.2) 60 (10.3) 202 (6.7) Neck 37 (5.7) 50 (4.8) 28 (3.7) 16 (2.7) 131 (4.3) Leg/foot 42 (6.5) 32 (3.1) 46 (6.1) 38 (6.5) 158 (5.2) Abdomen 21 (3.2) 5 (1.0) 30 (4.0) 1 (0) 57 (1.9) Buttocks/groin 7 (1.1) 8 (1.0) 9 (1.2) 0 (1.0) 30 (1.0) Lesion duration (% of answers in column) \3 mo 20 (3.1) 112 (10.8) 20 (2.7) 43 (7.4) 195 (16.5) 3-12 mo 89 (13.7) 327 (31.6) 113 (15.0) 197 (33.8) 726 (24.0) 1-2 y 50 (7.7) 209 (20.2) 113 (15.0) 112 (19.2) 484 (16.0) 2-5 y 86 (13.2) 166 (16.1) 148 (19.7) 107 (18.4) 507 (11.8) 5-10 y 29 (4.5) 63 (6.1) 64 (8.5) 38 (6.5) 194 (6.4) y 47 (7.2) 50 (4.8) 89 (11.8) 28 (4.8) 214 (7.1) Since birth 31 (4.8) 7 (1.0) 30 (4.0) 7 (1.2) 75 (2.5) Other 299 (45.9) 100 (9.7) 176 (23.4) 51 (8.7) 626 (20.7) Symptoms* (% of answers in column) None 409 (62.8) 307 (29.7) 388 (51.5) 196 (33.6) 1300 (43.0) Size change 113 (17.4) 320 (31.0) 192 (25.5) 142 (24.4) 767 (25.4) Itching 101 (15.5) 258 (25.0) 176 (23.4) 179 (30.7) 714 (23.6) Bleeding 57 (8.8) 273 (26.4) 43 (5.7) 104 (17.8) 477 (15.8) Tenderness 48 (7.4) 282 (27.3) 50 (6.6) 151 (25.9) 531 (17.6) Other 40 (6.1) 52 (5.0) 48 (6.4) 41 (7.0) 181 (6.0) Burning 13 (2.0) 49 (4.7) 2 (0) 19 (3.3) 83 (2.8) *n = Multiple symptoms possible; therefore percentages exceed 100. Table III. Histopathologic diagnoses for biopsied lesions No. (%) Histopathologic diagnosis N = 1685 Basal cell carcinoma 410 (24.3) Squamous cell carcinoma 240 (14.2) Benign keratoses 223 (13.2) Dysplastic nevus 154 (9.1) Premalignant/nonmelanocytic lesion 145 (8.6) (actinic keratoses) Benign nevus 138 (8.2) Cyst 73 (4.3) Melanoma 41 (2.4) Benign appendageal tumor 35 (2.1) Lentigo 29 (1.7) Benign vascular or lymphatic tumor 26 (1.5) Benign fibrohistiocytic lesion 24 (1.4) Neurofibroma 18 (1.1) Chondrodermatitis nodularis helicis 10 (0.6) Wart 9 (0.5) Other diagnoses 110 (6.5) agreement among nonbiopsied pigmented lesions. The addition of PLD images yielded similar agreement rates to those from macro images alone; the only significant difference was for biopsied pigmented lesions (P =.0152). Further, the addition of CID images significantly increased the agreement for pigmented lesions (both biopsied and nonbiopsied; P values \.034) compared with macro images alone. Kappa statistics were substantial to almost perfect. Management agreement For standard macro images, overall agreement between teledermatologists and clinic dermatologists in specified management ranged from 66.7% to 85.3% with the agreement for nonbiopsied nonpigmented lesion substantially higher (P values\.0001) than the rates of agreement in the other lesion categories. Rates of agreement with the addition of PLD images were comparable with those with just the macro

6 JAM ACAD DERMATOL VOLUME 72, NUMBER 3 Warshaw, Gravely, and Nelson 431 Table IV. Diagnostic and management plan agreement by lesion category Lesion category Image type No. of lesions Percent agreement (95% CI) Kappa statistic (95% CI) Macro Macro 1 PLD Macro 1 CID Macro vs macro 1 PLD Pairwise comparisons P values (Student t test) Macro vs macro 1 CID PLD vs macro 1 CID Nonbiopsied pigmented lesions N = 753 N = 752 N = 684 Primary diagnosis 75.7 ( ) 75.3 ( ) 80.1 ( ) *.0041* 0.56 ( ) 0.56 ( ) 0.62 ( ) Aggregated 91.0 ( ) 91.6 ( ) 93.9 ( ) *.0268* diagnoses 0.86 ( ) 0.87 ( ) 0.90 ( ) Management plan 71.1 ( ) 71.0 ( ) 79.8 ( ) \.0001* \.0001* 0.21 ( ) 0.19 ( ) 0.26 ( ) Biopsied pigmented lesions N = 651 N = 652 N = 595 Primary diagnosis 52.8 ( ) 53.4 ( ) 60.0 ( ) *.0045* 0.44 ( ) 0.45 ( ) 0.52 ( ) Aggregated 85.1 ( ) 88.8 ( ) 89.4 ( ).0152*.0337*.8991 diagnoses 0.84 ( ) 0.88 ( ) 0.88 ( ) Management plan 66.7 ( ) 69.6 ( ) 69.4 ( ) ( ) 0.33 ( ) 0.35 ( ) Nonbiopsied nonpigmented lesions N = 583 N = 579 Primary diagnosis 51.5 ( ) 50.3 ( ) n/a.5473 n/a n/a 0.38 ( ) 0.38 ( ) Aggregated 78.6 ( ) 79.1 ( ).8417 diagnoses 0.77 ( ) 0.77 ( ) Management plan 72.0 ( ) 72.0 ( ) ( ) 0.38 ( ) Biopsied nonpigmented lesions N = 1034 N = 1020 Primary diagnosis 45.7 ( ) 50.1 ( ) n/a.0046* n/a n/a 0.32 ( ) 0.37 ( ) Aggregated 85.4 ( ) 87.3 ( ).1144 diagnoses 0.84 ( ) 0.86 ( ) Management plan 85.3 ( ) 86.1 ( ) ( ) 0.41 ( ) CI, Confidence interval; CID, contact immersion dermoscopy; n/a, not applicable; PLD, polarized light dermoscopy. *Statistically significant. images. The addition of CID images increased the rates of agreement for nonbiopsied pigmented lesions (P values \.0001). Kappa statistics were fair. Agreement rate by teledermatologist s confidence For every lesion type and image package, agreement rates increased when confidence increased from low to moderate and from moderate to high (Table V). For primary diagnostic agreement (all image types), agreement rates were almost double or greater for cases where teledermatologists indicated high confidence in their diagnosis compared with those rated as low confidence. The aggregated diagnostic and management agreement rates for cases where teledermatologists indicated high confidence were at least 14 percentage points greater than those rated as low confidence; these differences were higher for nonpigmented lesions than pigmented lesions. Kappa statistic differences were similar. Agreement rate by teledermatologist-rated image quality For every lesion type and image package, agreement rates increased when image quality increased from low to moderate and from moderate to high

7 432 Warshaw, Gravely, and Nelson JAM ACAD DERMATOL MARCH 2015 Table V. Agreement rate by teledermatologist-rated confidence Image type Outcome Lesion category Confidence level No. of lesions Percent agreement (95% CI) Kappa statistic (95% CI) Low Moderate High Macro images N = 304 N = 1430 N = 1287 Primary diagnostic agreement Pigmented 41.7 ( ) 57.8 ( ) 77.6 ( ) 0.31 ( ) 0.47 ( ) 0.64 ( ) Nonpigmented 29.1 ( ) 43.0 ( ) 58.2 ( ) 0.20 ( ) 0.31 ( ) 0.49 ( ) Aggregated diagnostic agreement Pigmented 78.0 ( ) 86.4 ( ) 92.3 ( ) 0.77 ( ) 0.85 ( ) 0.89 ( ) Nonpigmented 69.8 ( ) 81.6 ( ) 87.9 ( ) 0.68 ( ) 0.80 ( ) 0.87 ( ) Management plan agreement Pigmented 56.1 ( ) 64.6 ( ) 76.3 ( ) 0.12 ( ) 0.30 ( ) 0.47 ( ) Nonpigmented 60.5 ( ) 80.6 ( ) 85.7 ( ) 0.19 ( ) 0.36 ( ) 0.52 ( ) Macro 1 PLD N = 257 N = 1254 N = 1343 Primary diagnostic agreement Pigmented 33.7 ( ) 54.0 ( ) 78.1 ( ) 0.17 ( ) 0.43 ( ) 0.68 ( ) Nonpigmented 27.5 ( ) 44.0 ( ) 61.9 ( ) 0.19 ( ) 0.33 ( ) 0.53 ( ) Aggregated diagnostic agreement Pigmented 74.4 ( ) 87.7 ( ) 94.4 ( ) 0.74 ( ) 0.86 ( ) 0.92 ( ) Nonpigmented 64.9 ( ) 82.8 ( ) 90.6 ( ) 0.64 ( ) 0.81 ( ) 0.90 ( ) Management plan agreement Pigmented 57.0 ( ) 64.3 ( ) 75.9 ( ) 0.16 ( ) 0.31 ( ) 0.50 ( ) Nonpigmented 62.0 ( ) 78.9 ( ) 87.7 ( ) 0.19 ( ) 0.36 ( ) 0.54 ( ) Macro 1 CID* N = 75 N = 456 N = 678 Primary diagnostic agreement 30.7 ( ) 61.0 ( ) 80.7 ( ) 0.23 ( ) 0.51 ( ) 0.72 ( ) Aggregated diagnostic agreement 72.0 ( ) 90.4 ( ) 94.5 ( ) 0.70 ( ) 0.88 ( ) 0.93 ( ) Management plan agreement 58.7 ( ) 70.2 ( ) 79.2 ( ) 0.13 ( ) 0.42 ( ) 0.54 ( ) CI, Confidence interval; CID, contact immersion dermoscopy; PLD, polarized light dermoscopy. *Pigmented lesions only. (Table VI). For cases where the teledermatologist rated the image quality as high, the primary diagnostic agreement rates (all image types) increased by 24 percentage points or greater than those rated as low image quality. The agreement rates for aggregated diagnoses and management agreement rates for cases where the teledermatologist rated the image quality as high were at least 10 points greater than the corresponding rates among cases where the teledermatologist rated the image quality as low. Image quality seemed to have less effect on aggregated diagnostic agreement rates for pigmented lesions than nonpigmented lesions, whereas the opposite was true for management agreement rates. Kappa statistic differences were similar. Association between teledermatologist-rated image quality and confidence There was a statistically significant association between teledermatologist s rated image quality and confidence level (P \.0001).

8 JAM ACAD DERMATOL VOLUME 72, NUMBER 3 Warshaw, Gravely, and Nelson 433 Table VI. Agreement by teledermatologist-rated image quality Image type Outcome Lesion category Image quality No. of lesions Percent agreement (95% CI) Kappa statistic (95% CI) Low Moderate High Standard macro images N = 556 N = 1836 N = 629 Primary diagnostic agreement Pigmented 52.3 ( ) 62.9 ( ) 82.2 ( ) 0.42 ( ) 0.53 ( ) 0.57 ( ) Nonpigmented 38.1 ( ) 45.7 ( ) 62.6 ( ) 0.27 ( ) 0.35 ( ) 0.54 ( ) Aggregated diagnostic agreement Pigmented 83.0 ( ) 87.9 ( ) 93.5 ( ) 0.81 ( ) 0.86 ( ) 0.87 ( ) Nonpigmented 75.6 ( ) 82.1 ( ) 92.3 ( ) 0.74 ( ) 0.81 ( ) 0.92 ( ) Management plan agreement Pigmented 58.5 ( ) 68.8 ( ) 78.4 ( ) 0.18 ( ) 0.38 ( ) 0.39 ( ) Nonpigmented 71.4 ( ) 81.9 ( ) 85.2 ( ) 0.29 ( ) 0.41 ( ) 0.50 ( ) Macro 1 PLD N = 422 N = 1646 N = 786 Primary diagnostic agreement Pigmented 43.8 ( ) 61.2 ( ) 79.5 ( ) 0.28 ( ) 0.51 ( ) 0.67 ( ) Nonpigmented 37.7 ( ) 48.5 ( ) 62.6 ( ) 0.28 ( ) 0.38 ( ) 0.54 ( ) Aggregated diagnostic agreement Pigmented 80.1 ( ) 89.9 ( ) 94.4 ( ) 0.78 ( ) 0.89 ( ) 0.92 ( ) Nonpigmented 68.5 ( ) 86.3 ( ) 90.9 ( ) 0.67 ( ) 0.85 ( ) 0.91 ( ) Management plan agreement Pigmented 56.8 ( ) 66.5 ( ) 80.2 ( ) 0.16 ( ) 0.34 ( ) 0.57 ( ) Nonpigmented 68.1 ( ) 83.4 ( ) 84.1 ( ) 0.27 ( ) 0.42 ( ) 0.50 ( ) Macro 1 CID* N = 95 N = 656 N = 458 Primary diagnostic agreement 44.2 ( ) 67.8 ( ) 78.8 ( ) 0.33 ( ) 0.59 ( ) 0.69 ( ) Aggregated diagnostic agreement 78.9 ( ) 90.9 ( ) 95.2 ( ) 0.76 ( ) 0.89 ( ) 0.94 ( ) Management plan agreement 63.2 ( ) 71.3 ( ) 81.4 ( ) 0.30 ( ) 0.43 ( ) 0.60 ( ) CI, Confidence interval; CID, contact immersion dermoscopy; PLD, polarized light dermoscopy. *Pigmented lesions only. DISCUSSION Main findings There are several important findings of this study. First, agreement was fair to substantial for primary diagnosis (45.7%-80.1%; kappa ), substantial to almost perfect for aggregated diagnoses (78.6%-93.9%; kappa ), and fair for management (66.7%-86.1%; kappa ). Second, diagnostic agreement (primary and aggregated) was highest for nonbiopsied pigmented lesions whereas management agreement was highest for biopsied nonpigmented lesions. Finally, although generally only modest improvements in agreement rates were observed with the addition of PLD images, CID significantly improved rates for pigmented lesions. Comparison of our results to other studies A recent systematic review on teledermatology 1,2 found a weighted average for primary diagnostic

9 434 Warshaw, Gravely, and Nelson JAM ACAD DERMATOL MARCH 2015 agreement of 62.3% from 5 lesion studies 3-7 and 1 dermoscopy pigmented lesion study 8 (n = 708 lesions). The overall primary diagnostic agreement rate for our study involving 3021 lesions was lower, 55.5%. For aggregated diagnostic agreement, a weighted average from 4 lesion studies 4-7 was 64.4% 1 (n = 358 lesions); the overall rate for our study involving 3021 lesions was higher, 85.5%. For management agreement, 2 store and forward teledermatology studies evaluated agreement of the triage management decision of refer or not refer for pigmented lesions 15 or skin lesions 3 ; percent concordance was 80% 15 and 61% 3 for these 2 studies, yielding a weighted average of 75.3% (809/1075). 1 Two lesion studies evaluated agreement for the diagnostic procedure decision biopsy versus no biopsy and found concordance rates of 100% 16 and 95% 7 (weighted average of 98.5%, 68/69). 1 One pigmented lesion study found an agreement rate of 96% for 3 different management options. 17 Our study had 5 possible management choices and a write-in option that resulted in a total of 11 management categories. This larger number of management choices (11 vs 2-3) may explain our lower rate of agreement (75.2%) than previous studies. Kappa statistics control for agreement caused by chance alone. Because of the smaller number of management options, chance agreement was likely higher; this may explain lower kappa statistic values. 18 Dermoscopy Although PLD increased all 3 agreement outcomes over macro images, this increase was modest (# 4.4 percentage points) and only significantly better in 2 categories (aggregated diagnostic agreement for biopsied pigmented lesions and primary diagnostic agreement for biopsied nonpigmented lesions). The improvement in agreement outcomes was higher for CID (# 8.7 percentage points) as compared with macro images alone (all statistically significant except for management agreement for biopsied pigmented lesions). Importantly, all of the teledermatologists in our study were experts in dermoscopy. 14 This study confirms that adding dermoscopic images increases agreement, as it does in the clinical setting. 19 This underscores the need for proper equipment and dermoscopy-trained teledermatologists in the clinical setting of teledermatology involving pigmented lesions. Confidence and image quality It is not surprising that high confidence of teledermatologists was associated with better agreement rates. This likely reflects the difference between easy and hard lesions. It is also not surprising that high-quality images would result in higher diagnostic agreement rates. Importantly, our previous analysis of biopsied lesions in this cohort found that the accuracy (using gold standard of histopathology) was significantly lower for teledermatologists as compared with clinic dermatologists and several melanomas were mismanaged, even with high levels of confidence and high-quality photographs. 11 Limitations There are several important limitations to this study. First, the study population was primarily male and Caucasian. Second, although teledermatologists were blinded to the study purpose, they were aware that it was a study and therefore may not have been as careful or conservative in generating diagnoses and management plans as in a real-world setting. Third, pigmented biopsied lesions were oversampled to meet recruitment goals; other categories were approached in a convenience fashion. Fourth, although interrater reliability was evaluated, 13 intrarater reliability of teledermatologists was not. Finally, this study was not designed to address other important outcomes such as cost, satisfaction, or clinical outcomes. Conclusions In summary, this study of store and forward teledermatology involving over 3000 skin neoplasms found moderate to almost perfect diagnostic agreement whereas management agreement was fair. CID significantly improved rates for pigmented lesions. Further studies are needed to assess clinical outcomes, patient and provider satisfaction, and costeffectiveness in managing not only skin neoplasms but other categories of dermatologic conditions. REFERENCES 1. Warshaw EM, Hillman YJ, Greer NL, et al. Teledermatology for diagnosis and management of skin conditions: a systematic review. J Am Acad Dermatol. 2011;64: Warshaw E, Greer N, Hillman Y, et al. Teledermatology for diagnosis and management of skin conditions: a systematic review of the evidence. VA-ESP Project #09-009; Available from; URL: publications/esp/telederm.cfm. Accessed November 20, Bowns IR, Collins K, Walters SJ, McDonagh AJ. Telemedicine in dermatology: a randomized controlled trial. Health Technol Assess. 2006;10: Oakley AM, Reeves F, Bennett J, Holmes SH, Wickham H. Diagnostic value of written referral and/or images for skin lesions. J Telemed Telecare. 2006;12: Mahendran R, Goodfield MJ, Sheehan-Dare RA. An evaluation of the role of a store-and-forward teledermatology system in

10 JAM ACAD DERMATOL VOLUME 72, NUMBER 3 Warshaw, Gravely, and Nelson 435 skin cancer diagnosis and management. Clin Exp Dermatol. 2005;30: Barnard CM, Goldyne ME. Evaluation of an asynchronous teleconsultation system for diagnosis of skin cancer and other skin diseases. Telemed J E Health. 2000;6: Whited JD, Mills BJ, Hall RP, Drugge RJ, Grichnik JM, Simel DL. A pilot trial of digital imaging in skin cancer. J Telemed Telecare. 1998;4: Piccolo D, Smolle J, Wolf IH, et al. Face-to-face diagnosis vs telediagnosis of pigmented skin tumors: a teledermoscopic study. Arch Dermatol. 1999;135: Austin Data Base, Fiscal Year [VA-specific database.] 10. Warshaw EM, Lederle FA, Grill JP, et al. Accuracy of teledermatology for nonpigmented neoplasms. J Am Acad Dermatol. 2009;60: Warshaw EM, Lederle FA, Grill JP, et al. Accuracy of teledermatology for pigmented neoplasms. JAmAcadDermatol. 2009;61: Warshaw EM, Gravely AA, Nelson DB. Accuracy of teledermatology/teledermatoscopy and clinical-based dermatology for specific categories of skin neoplasms. J Am Acad Dermatol. 2010;63: Warshaw EM, Gravely AA, Bohjanen KA, et al. Interobserver accuracy of store and forward teledermatology for neoplasms. J Am Acad Dermatol. 2010;62: Warshaw EM. Reply. J Am Acad Dermatol. 2009;61: Jolliffe VM, Harris DW, Morris R, Wallace P, Whittaker SJ. Can we use video images to triage pigmented lesions? Br J Dermatol. 2001;145: Shapiro M, James WD, Kessler R, et al. Comparison of skin biopsy triage decisions in 49 patients with pigmented lesions and skin neoplasms: store-and forward teledermatology vs face-to-face dermatology. Arch Dermatol. 2004;140: Di Stefani A, Zalaudek I, Argenziano G, Chimenti S, Soyer HP. Feasibility of a two-step teledermatologic approach for the management of patients with multiple pigmented skin lesions. Dermatol Surg. 2007;33: McHugh ML. Interrater reliability: the kappa statistic. Biochem Med. 2012;22: Kittler H, Pehamberger H, Wolff K, Binder M. Diagnostic accuracy of dermoscopy. Lancet Oncol. 2002;3: