Test Retest and Between-Site Reliability in a Multicenter fmri Study

Transcription

1 Human Bain Mapping 29: (2008) Test Retest and Between-Site Reliability in a Multicente fmri Study Lee Fiedman, 1 * Hal Sten, 2 Gegoy G. Bown, 3 Daniel H. Mathalon, 4 Jessica Tune, 1 Gay H. Glove, 5 Randy L. Gollub, 6,7 John Lauiello, 8 Kelvin O. Lim, 9 Tyone Cannon, 10 Douglas N. Geve, 7 Heny Jeemy Bockholt, 11 Aysenil Belge, 12,13 Byon Muelle, 9 Michael J. Doty, 14 Jianchun He, 15 William Wells, 16 Padhaic Smyth, 17 Steve Piepe, 18 Seyoung Kim, 17 Maek Kubicki, 19 Mak Vangel, 6,20 andsteveng.potkin 1 1 Depatment of Psychiaty and Human Behavio, Univesity of Califonia Ivine, Ivine, Califonia 2 Depatment of Statistics, Univesity of Califonia Ivine, Ivine, Califonia 3 Psychology Sevice 116B, Univesity of Califonia San Diego, VA San Diego Healthcae System, San Diego, Califonia 4 Psychiaty Sevice 116A, VA Connecticut Healthcae System, Connecticut 5 Depatment of Radiology, Stanfod Univesity, Stanfod, Califonia 6 Depatment of Psychiaty, Massachusetts Geneal Hospital, Chalestown, Massachusetts 7 Matinos Cente fo Biomedical Imaging, Massachusetts Geneal Hospital, Chalestown, Massachusetts 8 Depatment of Psychiaty, Univesity of New Mexico, Albuqueque, New Mexico 9 Depatment of Psychiaty, Univesity of Minnesota, Minneapolis, Minnesota 10 Depatment of Psychology, Los Angeles, Califonia 11 Mophomety and Neuoinfomatics Coe, The MIND Institute, 1101 Yale Boulevad NE, Albuqueque, New Mexico 12 Duke-UNC Bain Imaging and Analysis Cente, Duke Univesity Medical Cente, Duham, Noth Caolina 13 Depatment of Psychiaty, Univesity of Noth Caolina at Chapel Hill, Chapel Hill, Noth Caolina 14 Biomedical Engineeing Coe, The MIND Institute, 1101 Yale Boulevad NE, Albuqueque, New Mexico 15 Depatment of Psychiaty, Univesity of Iowa Hospital and Clinic, Iowa City, Iowa 16 Depatment of Radiology, Havad Medical School and Bigham and Women s Hospital, Boston, Massachusetts 17 Depatment of Compute Science, Univesity of Califonia Ivine, Ivine, Califonia 18 Isomics, Inc., 203 Fanklin Steet, Cambidge, Massachusetts 19 Laboatoy of Neuoscience, Depatment of Psychiaty, Havad Medical School, Bockton, Massachusetts 20 MGH/MIT GCRC Biomedical Imaging Coe, Chalestown, Massachusetts Contact gant sponsos: National Cente fo Reseach Resouces (NCRR), National Institutes of Health (NIH); Contact gant numbe: 1 U24 RR *Coespondence to: Lee Fiedman, 1312 Michael Hughes D. NE, Albuqueque, NM 87112, USA. lfiedman10@comcast.net Received fo publication 20 June 2006; Accepted 23 May 2007 DOI: /hbm Published online 17 July 2007 in Wiley InteScience (www. intescience.wiley.com). VC 2007 Wiley-Liss, Inc.

2 Test Retest and Between-Site Reliability Abstact: In the pesent epot, estimates of test etest and between-site eliability of fmri assessments wee poduced in the context of a multicente fmri eliability study (FBIRN Phase 1, Five subjects wee scanned on 10 MRI scannes on two occasions. The fmri task was a simple block design sensoimoto task. The impulse esponse functions to the stimulation block wee deived using an FIR-deconvolution analysis with FMRISTAT. Six functionally-deived ROIs coveing the visual, auditoy and moto cotices, ceated fom a pio analysis, wee used. Two dependent vaiables wee compaed: pecent signal change and contast-to-noise-atio. Reliability was assessed with intaclass coelation coefficients deived fom a vaiance components analysis. Test etest eliability was high, but initially, between-site eliability was low, indicating a stong contibution fom site and site-by-subject vaiance. Howeve, a numbe of factos that can makedly impove between-site eliability wee uncoveed, including inceasing the size of the ROIs, adjusting fo smoothness diffeences, and inclusion of additional uns. By employing multiple steps, between-site eliability fo 3T scannes was inceased by 123%. Dopping one site at a time and assessing eliability can be a useful method of assessing the sensitivity of the esults to paticula sites. These findings should povide guidance to othes on the best pactices fo futue multicente studies. Hum Bain Mapp 29: , VC 2007 Wiley-Liss, Inc. Key wods: test etest; epoducibility; intaclass coelation coefficient; multicente; FMRI INTRODUCTION Multicente fmri studies have a numbe of advantages, as outlined by Fiedman et al. [2006] and Fiedman and Glove [2006]. They also pose seious challenges. One common goal of most such studies is to liteally mege the data fom seveal scannes to incease the sample size applied to a substantive question of inteest, fo example, the elationship between cetain imaging phenotypes and genetic infomation. Liteally meging such data equies data fom diffeent scannes to be intechangeable and is only easonable if scanne diffeences in fmri esults can be minimized, i.e., the assessments fom one scanne to anothe ae eliable. 1 One appoach to assessing measuement eliability is to pefom a eliability study pio to a scientifically substantive study. If the between-site eliability is low, souces of uneliability might be identified and coected. Classically, eliability of this fom of data is assessed with an intaclass coelation coefficient, which can be estimated diectly fom an appopiate analysis of vaiance table o fom vaiance components [ICC; Conbach et al., 1972; Shout and Fleiss, 1979]. The ICC is a numbe that anges fom 0.0 to Meging of multisite data is not the only easonable appoach. One can, fo example, model the site effects. The simplest case of this would be to teat the site facto as a fixed effect and estimate the mean shift fom site to site. One could also model diffeent within-site vaiance estimates fo each site independently. Moe complex models ae also possible. We believe that stating out with the meging stategy povides a basis fo the discovey of factos that might attenuate between-site eliability. Futhemoe, we believe that thee ae a numbe of advantages to modeling site as a andom effect, if possible. Cicchetti and Spaow [1981] [Cicchetti, 2001] pesented guidelines fo intepetation of ICCs as follows: poo (below 0.40), fai ( ), good ( ), and excellent (above 0.75). The ICC is a commonly used metic to assess test etest eliability in fmri [Aon et al., 2006; Kong et al., 2007; Manoach et al., 2001; Specht et al., 2003; Wei et al., 2004], although othe metics have been employed [Genovese et al., 1997; Le and Hu 1997; Liou et al., 2006; Maita et al., 2002; Zou et al., 2005]. Thee ae seveal types of ICC [Shout and Fleiss, 1979] but if the goal is to liteally mege data fom sites, then the choice is naowed to ICC (type 2, 1) in the nomenclatue of Shout and Fleiss [1979] (see below). This ICC measues the degee of absolute ageement of each ate (scanne) with each othe ate. If this is sufficiently lage, then one has a good basis fo meging data. We pesent the esults of vaiance components analyses and associated ICCs fo a multisite study pefomed by the Function Biomedical Infomatics Reseach Netwok (fbirn). The fbirn goup published a pevious aticle on these data showing the impact of scanne site, task un, and test occasion on functional magnetic esonance imaging esults [Zou et al., 2005]. Howeve, this initial study was not aimed at measuing the magnitude of the vaious souces of unwanted vaiance in multisite fmri data. In this aticle, we use vaiance components analysis to measue the magnitude of the vaiance components associated with scanne site, task un, and testing occasion on the amplitude of the fmri signal [Dunn, 2004]. The pesent study tests the following hypotheses: Hypothesis 1. The esults of fmri studies can be expessed in seveal ways, fo example, pecent signal change (PSC), 959

3 Fiedman et al. t-value, P-value, egession b-weight, Peason coelation coefficient, location of activation, numbe of activated voxels, etc. In the pesent study, we hypothesized that noise would be uneliable acoss sites and that theefoe measues, which have an estimate of noise in the denominato (CNR-type measues), would have lowe eliability than measues of signal magnitude only (PSC). This hypothesis was stimulated patly by the epot of Cohen and DuBois [1999], who noted that PSC povided moe eliable estimates than numbe of activated voxels. Hypothesis 2. Measues that ae based on the median value fom an ROI will be moe eliable than measues that ae based on the maximum value fom an ROI. This was based on the notion that the maximum value could be some sot of outlie o unusual value, pehaps highly influenced by local venous achitectue (especially at 1.5T) [Ugubil et al., 1999]. Hypothesis 3. In a pevious epot [Fiedman et al., 2006], we found that image smoothness (measued as a FWHM) was elated to task effect size. 2 In that epot, we found that a majo cause of site diffeences in image smoothness was the pesence and type of apodization (k-space) filte employed duing image econstuction. We hypothesize that adjusting fo smoothness diffeences between scannes will incease eliability estimates fo CNR type measues. Hypothesis 4. The size of an ROI could well have an impact on the eliability of the measues taken fom it. If an ROI is too small, it may not captue the pimay activation fom vaious sites, especially when the same ROI is used acoss field stength. It is known that geometic distotion of functional images is geate at 3.0T than at 1.5T such diffeences could lead to somewhat diffeent esults if the same ROI is used fo both. On the othe hand, an ROI that is too lage will have low anatomic specificity. In the pesent study, fou uns of a sensoimoto task wee collected in each visit. This allowed us to compae the eliability of the aveage of two, thee, and fou uns to a single un. Theoetically, as is clea fom ou fomulae (see below), eliability will incease with moe uns. This is analogous to esults fom classical test theoy, in which inceasing the numbe of test items will incease eliability of the test (Speaman-Bown Pophecy fomula [Lod and 2 The statistical use of the tem effect size is diffeent fom the imaging use of the tem. In statistics, effect size efes to a quantity computed as a measue of effect magnitude (the numeato) divided by a measue of esidual vaiance o eo vaiance (the denominato). Fo example, fo a two-sample test of means, Cohen s D is the diffeence in means divided by the pooled standad deviation. In imaging, the tem effect size is typically used to descibe an effect magnitude (b-weight o pecent signal change) uncontolled (not divided) by an estimate of noise. Thoughout this manuscipt, we always use the tem in the statistical sense. Novick, 1968]). Howeve, factos such as inceasing fatigue and inattention ove time may educe the eliability of late uns. Theefoe, it was of inteest to see how well empiical esults match the theoetical expectation that aveaging uns inceases eliability. The pesent epot is based on the FBIRN Phase I taveling subject study [Fiedman and Glove, 2006; Fiedman et al., 2006; Zou et al., 2005]. In this study, five subjects wee scanned on 10 diffeent scannes on two occasions. This povided an excellent dataset with which to assess test etest and between-site eliability and to test ou hypotheses. MATERIALS AND METHODS Subjects Five healthy, English-speaking males (mean age: 25.2, ange ) paticipated in this study. All wee ighthanded, had no histoy of psychiatic o neuological illnesses and had nomal heaing in both eas. Each subject taveled to nine sites (10 scannes) (Table I), whee they wee scanned on two consecutive days fo a total of 20 scans pe paticipant. Thee wee no missing subject visits (scan sessions), i.e., all 100 visits (5 subjects 3 2 visits 3 10 scannes) wee available fo analysis. Howeve, one scan session (1%) was unusable fo technical easons. All subjects wee instucted to avoid alcohol the night befoe the study, caffeine 2 h pio to the study and to get a nomal night s sleep the night befoe a scan session. This study was conducted in compliance with the Code of Ethics of the Wold Medical Association (Declaation of Helsinki) and the Standads established by the Institutional Review Boad of each paticipating institution. Afte a full explanation of the pocedues employed, infomed consent was obtained fom evey subject befoe paticipation in this study and befoe evey scan session. Image Acquisition A bite ba was used to stabilize each subject s head and was placed in the subject s mouth at the beginning of each scan session. An initial T2-weighted, anatomical volume fo functional ovelay was acquied fo each subject (fast spin-echo, tubo facto 5 12 o 13, oientation: paallel to the AC-PC line, numbe of slices 5 35, slice thickness 5 4 mm, no gap, TR 5 4,000 ms, TE 5 68, FOV 5 22 cm, matix , voxel dimensions mm mm 3 4 mm). The paametes fo this T2 ovelay scan wee allowed to vay slightly fom scanne to scanne accoding to field stength o othe local technical factos. The anatomical scan was followed eithe by a woking memoy task (total of 14.7 min) o an attention task (total of 16 min). Thee subjects always pefomed the woking memoy task and two subjects always pefomed the attention task. Ove the next hou o so, subjects pefomed fou uns of a sensoimoto task (descibed below, 4.25 min pe un, total of 17 min), two uns of a beath-hold 960

4 Test Retest and Between-Site Reliability TABLE I. Desciption of hadwae and sequences of the nine sites (10 scannes) paticipating in this study, five 1.5T scannes, fou 3T scannes, and one 4T scanne Site code Abbeviation Field stength (T) Manufactue RF coil type Functional sequence 1 SITE GE Nvi LX TR quadatue head Spial 2 SITE GE Signa CV/i TR quadatue head EPI 3 SITE Siemens Sonata RO quadatue head EPI 4 SITE Philips/Picke RO quadatue head EPI 5 SITE Siemens Symphony TR quadatue head EPI 6 SITE GE GE TR eseach coil EPI 7 SITE Siemens Tio TR quadatue head EPI -Dual Echo 8 SITE Siemens Tio TR quadatue head EPI 9 SITE GE CV/NVi Elliptical quadatue head Spial in/out 10 SITE GE Nvi LX TR quadatue head Spial task (4.25 min), and two esting state scans (fixation on a cosshais, 4.25 min). These eight scans wee pefomed in a countebalanced ode. The entie scan session was epeated the following day. In the pesent epot, only data fom the fou sensoimoto uns is pesented. The functional data wee collected using echo-plana (EPI) tajectoies (seven scannes) o spial tajectoies (thee scannes) (Table I) (oientation: paallel to the AC- PC line, numbe of slices 5 35, slice thickness 5 4 mm, no gap, TR sec, TE 5 30 ms on the 3T and 4T scannes, 40 ms on the 1.5T scannes, FOV 5 22 cm, matix , voxel dimensions mm mm 3 4 mm). SITE 7 employed a double echo EPI sequence and SITE 9 employed a spial in/spial out sequence. All the spial acquisitions wee collected on Geneal Electic (GE) scannes. The sensoimoto task poduced fou uns of 85 volumes each (85 TRs). The RF coils used vaied with each scanne (Table I). Sensoimoto Task The sensoimoto (SM) task was designed initially fo calibation puposes and employed a block design, with each block taking 10 TRs (30 sec) beginning with five TRs (15 s) of est (subject instucted to stae at fixation coss) and five TRs (15 s) of sensoimoto activity (see below). Thee wee eight full cycles of this followed by a five TR est peiod at the end fo a total of 85 TRs (4.25 min). Duing the active phase, subjects wee instucted to tap thei finges bilateally in synchony with binaual tones, while watching an altenating contast checkeboad. The checkeboad flash and tone pesentation wee simultaneous. The subjects wee instucted to tap thei finges in an altenating finge tapping patten (index, middle, ing, little, little, ing, middle, index, index...) in synchony with the tones and checkeboad flashes. The thumb was not used in this study. Each tone was 166 ms long with 167 ms of silence. The tone sequence utilized a dissonant seies geneated by a synthesize (Midi notes 60, 64, 68, 72, 76, 80, 84, 88, 86, 82, 78, 74, 70, 66, 62, 58). The subjects esponses wee ecoded and monitoed with the PST Seial Response Box (Psychology Softwae Tools, Pittsbugh, PA). Pepocessing fmri Data Analysis The fist step of image pocessing was accomplished using Analysis of Functional NeuoImage (AFNI) softwae [Cox, 1996]. The fist two volumes wee discaded to allow fo T1-satuation effects to stabilize. All lage spikes in the data wee emoved fom each sensoimoto un, and each un was motion-coected (i.e., spatially egisteed to the middle volume of the un, TR 5 42). The data wee then slice-time-coected. A mean functional (T2*) image was ceated. The mean T2* image fo each un was spatially nomalized to an EPI canonical image in MNI space using tools available in SPM5 ( This included affine tansfomations and thee nonlinea iteations. We limited the nonlinea iteations to thee to contol the amount of defomation. The spatial tansfomations wee applied to the time seies data as well, and the time seies was esampled at a mm 3 voxel size. Measuing PSC and CNR The impulse esponse functions (IRF) fo each voxel wee estimated using Keith Wosley s package, FMRISTAT [Wosley et al., 2002] ( fmistat/), accoding to the FIR-Deconvolution method outlined at the FMRISTAT web page. The IRFs wee 30 s long (10 time points, 3 s apat), coveing the duation of the on and off block peiods. Tempoal dift was emoved by adding a linea and a quadatic component to the model. The PSC IRFs wee calculated by dividing the estimates (bs) by the model intecept (mean baseline level) and multiplying by 100. The CNR IRFs wee the t-values fo each time point in the FIR model. Image data tansfe between pogams (AFNI, SPM5, and FMRISTAT) was geatly facilitated by the application of the new NifTi standad ( nifti-1/). ROIs The ROIs ae displayed in Figue 1. The ROIs include only voxels that wee activated at all 10 scannes with an 961

5 Fiedman et al. factos fo this analysis, and fixed-effects included fieldstength, ROI, and the field-stength by ROI inteaction. The fixed-effects wee tested using planned contasts between field stength fo all ROIs togethe and each ROI sepaately. Planned contasts included a test of 3T > 1.5T. (The 4T > 3T test was not pefomed since thee was only one site at 4T.) Since these contasts wee obvious diectionally specific hypotheses, the P-values povided ae one-tailed. The multiple ROI contasts wee contolled fo the false discovey ate (FDR) [Benjamini and Hochbeg, 1995] using SAS PROC MULTTEST. Cohen s D values ae also povided to compae effect sizes. Effect sizes fo the field-stength effect fom PSC and CNR wee compaed using a paied t-test. Measuing eliability Vaiance components wee estimated fo each scala using SAS PROC Vacomp (SAS, Cay, NC), which employed the esticted maximum likelihood (REML) method. The vaiance components ae estimated accoding to the following model: Figue 1. ROIs. Fou of six ROIs employed in this study. ROIs ae shown in white. Fo each ROI, an axial, coonal, and sagittal view is pesented. The emaining two ROIs (ight moto and ight auditoy) wee compaable to the contalateal ROIs shown in this figue. uncoected P value < and in all five subjects with an uncoected P value < Thee wee six ROIs identified: left and ight moto cotex (LM and RM), left and ight auditoy cotex (LA and RA), bilateal visual cotex (BV), and the bilateal supplementay moto aea (SM). Extacting Scala Values fom Each ROI Fou IRFs wee extacted fom each ROI: (1) median IRF in pecent change units, (2) maximum IRF in pecent change units, (3) median IRF in CNR units, and (4) maximum IRF in CNR units. To aive at a scala value fom each IRF (fo each un of the SM task) the peak value of each IRF fom 6 s postblock onset to 18 s postblock onset was obtained. Statistical Analysis Data wee available fo five subjects at 10 scannes. Each subject was scanned on two visits, and thee wee fou SM uns pe visit. Testing field-stength effects To test fo field-stength effects, a mixed-model ANOVA was caied out fo median PSC and median CNR. The andom factos noted below wee included as andom Y ijkl ¼ mean þ subject i þ site j þ site-by-subject ij þ visitðsite-by-subjectþ ijk þ unexplained ijkl with Y ijkl denoting the dependent measue fo subject i, site j, visit k, and un l. Each facto was teated as a andom effect. (The site facto is often thought of as a fixed effect, but in the context of a multicente study it is desiable to think about a population of potential sites.) In this fomulation, visits ae teated as nested within site-subject combinations and the esidual tem is an estimate of the vaiance fo uns nested within subject-by-site-by-visit. This fomulation models visits and the uns that occu on these visits as distinct measuement occasions. The model also allows fo possible day-to day vaiation in magnet pefomance. As discussed below, altenative models ae possible and do not appeciably change the eliability esults. 3 3 The pesent data set allows us to conside the effect on eliability estimates of vaying the statistical model used to estimate the vaiance components. Fo example, one might view data fom the cuent study as being geneated by a completely cossed statistical design, i.e., visit cossed with site, subject and thei inteaction. We did exploe the fully cossed model, still teating all factos as andom. The effects of visit and un ae extemely small as ae all of thei inteactions except fo the inteaction visit 3 subj 3 site, which is essentially equivalent to ou nested vaiance component fo visit and the full inteaction, which is essentially ou esidual vaiance component. Although the analysis based on a model with visit and un viewed as factos cossed with site and subject does poduce slightly diffeent eliability values, these values do not vay geatly fom those epoted fo the nested design. One might expect a bigge diffeence between the two appoaches if thee was a moe consistent patten acoss visits o un (e.g., leaning o habituation effects). 962

6 Test Retest and Between-Site Reliability TABLE II. Vaiance components and eliability estimates fo PSC ROI extaction method Field Region Site Subject Site 3 Subject Visit Unexplained ICC_BET a ICC_T-R b Median 1.5 BV LA LM RA RM SM BV LA LM RA RM SM Maximum 1.5 BV LA LM RA RM SM BV LA LM RA RM SM a ICC_BET 5 between-site ICC. b ICC_T-R 5 test etest ICC. Tables II and III pesents esults fom the vaiance components analysis of PSC (Table II) and CNR (Table III). Analyses ae caied out fo each of six bain egions, sepaately fo 1.5T and 3.0T imaging sites. Two diffeent measues of eliability ae developed and pesented in Tables II and III. To descibe these measues, we fist TABLE III. Vaiance components and eliability estimates fo CNR ROI extaction method Field (T) Region Site Subject Site 3 Subject Visit Unexplained ICC_BET a ICC_T-R b Median 1.5 BV LA LM RA RM SM BV LA LM RA RM SM Maximum 1.5 BV LA LM RA RM SM BV LA LM RA RM SM a ICC_BET 5 between-site ICC. b ICC_T-R 5 test-etest ICC. 963

7 Fiedman et al. Note that the between-site eliability coefficient povides an estimate of the eliability of an imaging paamete aveaged acoss fou uns fo a andomly selected subject studied at a single andomly selected site on one andomly selected occasion. A measue fo test etest eliability asks about the eliability (o coelation) of two visits fo the same subject at the same site but on diffeent days. This eliability is calculated as Test-Retest Reliability ¼ðVD subject þ VD site þ VD subject-by-siteþ=total Visit Vaiance The between-site eliability estimates ae analogous to ICC (type 2, 1) and the test etest eliability estimates ae analogous to ICC (type 1, 1) fom Shout and Fleiss [1979] but have been adapted to the moe complex design. RESULTS Field-Stength Effects (Fig. 2, Table IV) Figue 2. Field stength effects. (A) Mean PSC estimates (based on median ROI extaction) fo 1.5T, 3T, and the 4T scanne acoss six ROIs (BV5 bilateal visual cotex, LA 5 left auditoy cotex, LM 5 left moto cotex, RA 5 ight auditoy cotex, RM 5 ight moto cotex and SM 5 supplementay moto cotex). (B) Mean CNR estimates (based on median ROI extaction) fo 1.5T, 3T, and the 4T scanne acoss six ROIs. intoduce a total visit vaiance tem, which is the sum of all of the vaiance components associated with the above model: Total Visit Vaiance ¼ðVD subject þ VD site þ VD site-by-subject þ VD visit þðvd unexplained=4þþ whee VD stands fo vaiance due to. We divide VD_unexplained by 4 to get the total visit vaiance (aveage ove fou uns) athe than total vaiance fo a single un. A measue of between-site eliability (fo a visit consisting of fou uns) is the coelation of two measues (based on the mean of fou uns) fo the same subject but at diffeent sites. In tems of the vaiance components estimated as above (and pesented in Tables II and III) this is: Between-Site Reliability ¼ VD subject=total Visit Vaiance Median PSC and CNR both incease with field-stength (Fig. 2). The 3T scannes had highe median PSC than the 1.5T scannes (Fig. 2A, Table IV). The all-egion test was statistically significant as wee five of six ROI-specific tests. Contolling fo an FDR of 0.05, two of the ROI-specific tests wee statistically significant. The Cohen s D effect sizes wee vey lage. The 3T scannes had highe median CNR than the 1.5T scannes (Fig. 2B, Table IV). The allegion test was statistically significant as wee all of the ROI-specific tests. Contolling fo an FDR of 0.05, all of the ROI-specific tests wee statistically significant. The Cohen s D effect sizes wee also vey lage. Howeve, the effect sizes fo median CNR wee statistically significantly highe than the effect sizes fo median PSC (P , two-tailed, paied t-test). The pesence of consistent mean elevations fo the 4T scanne compaed to the 3T scannes (11 of 12 measues), indicates that including the 4T scanne with the 3T scannes will lowe eliability. Fo this eason, we chose not to lump the 4T in with the high field goup. Test Retest Reliability (Fig. 3A, Tables II and III) Test etest eliability was geneally high (Fig. 3A). Fo the median PSC measue, the median test etest eliability was 0.76 (25th pecentile , 75th pecentile ). Fo the median CNR measue, the median test etest eliability was 0.74 (25th pecentile , 75th pecentile ). Thus, the cental tendency of test-etest eliability is at the bode between good and excellent eliability. Between-Site Reliability (Fig. 3B, Tables II and III) Between-site eliability was much lowe than test etest eliability (Fig. 3B vesus Fig. 3A). Fo the median PSC measue, the median between-site eliability was 0.22 (25th 964

8 Test Retest and Between-Site Reliability TABLE IV. Field-stength effects Contast Region Estimate T df One-tailed P-value FDR P-value Cohen s D Pecent signal change 3 T > 1.5 T ALL T > 1.5 T BV T > 1.5 T LA T > 1.5 T LM T > 1.5 T RA T > 1.5 T RM T > 1.5 T SM Contast-to-noise atio 3 T > 1.5 T ALL T > 1.5 T BV T > 1.5 T LA T > 1.5 T LM T > 1.5 T RA T > 1.5 T RM T > 1.5 T SM pecentile , 75th pecentile ). Fo the median CNR measue, the median between-site eliability was 0.25 (25th pecentile , 75th pecentile ). Both ICCs can be descibed as poo. slope, each incease by 1 mm in smoothness (FWHM) is associated with a 0.03 decline in median PSC. With PSC estimates in the ange of , this amounts to between Compaing Median-based Measues to Maximum-based Measues on Between-Site Reliability (Fig. 4, Table V) Between-site eliability estimates fo median and maximum PSC (Fig. 4A) and CNR (Fig. 4B) measues ae summaized in Table V. Thee is no obvious winne in these compaisons, and Wilcoxon Paied Tests eveal no statistically significant diffeences. The maximum measues do have highe median eliability (Table V) but the patten is not consistent acoss egions and measues. Moeove, fo PSC fo the 1.5T_BV measue, whee eliability is geatest, thee is a maked dop in eliability estimate in going fom median to maximum measue. The Effect of Smoothness Adjustment on Between-Site Reliability (Fig. 5, Table VI) In a pevious epot [Fiedman et al., 2006; see also Lowe and Soenson, 1997], we found that image smoothness had a stong effect on activation effect size, i.e., the multiple R 2. This is most simila to the CNR measue used heein, although it would be bette elated to contast-tototal-tempoal-vaiance-atio. Having in ou possession smoothness estimates fom that pape and median PSC and CNR estimates hee, it was of inteest to detemine if pio adjustment fo smoothness diffeences would have a beneficial effect on between-site eliability. Median PSC and CNR measues wee egessed against aveage smoothness estimates (FWHM units) (Table VI). Geneally, thee was a negative slope fo the elationship between smoothness and median PSC. Although the effect is small, with a median slope of , the elationship was statistically significant fo six measues. With this Figue 3. Reliability. (A) Test etest eliability estimates fo PSC and CNR fo 1.5T and 3T scannes at six ROIs. See Figue 2 fo ROI definitions. (B) Between-site eliability estimates fo PSC and CNR fo 1.5T and 3T scannes at six ROIs. 965

9 Fiedman et al. Figue 4. Compaing the median and the maximum. (A) Between-site eliability estimates fo PSC based on eithe median ROI extaction o maximum ROI extaction fo 1.5T and 3T scannes at six ROIs. See Figue 2 fo ROI definitions. (B) Between-site eliability estimates fo CNR based on eithe median ROI extaction o maximum ROI extaction fo 1.5T and 3T scannes at six ROIs. Measue TABLE V. Between-site eliability estimates fo median-based and maximum-based measues ROI extaction method 25th pecentile Median 75th pecentile Pecent signal Median change Pecent signal Maximum change CNR Median CNR Maximum Figue 5. Effects of contolling fo smoothness. (A) The effect of pio adjustment fo smoothness on between-site eliability of median PSC at 1.5T and 3T fo six ROIs. See Figue 2 fo ROI definitions. (B) The effect of pio adjustment fo smoothness on between-site eliability of median CNR at 1.5T and 3T fo six ROIs. TABLE VI. Slope estimates fo the elationship between image smoothness and PSC o CNR Field Region PSC_Slope PSC P-value CNR_Slope CNR_P-value 1.5 BV LA LM RA RM SM BV LA LM RA RM SM

10 Test Retest and Between-Site Reliability The Effect of ROI Size on Between-Site Reliability (Fig. 6) One concen was that ou ROIs wee too small, even though they wee defined to include activations fom all sites (see above). All the oiginal ROIs wee dilated in the x, y, and z diection by thee voxels. The dilated ROIs poduced statistically significant inceases in between-site ICC fo PSC (Wilcoxon Test, P ), especially at 3T (impovement at 1.5T 5 47%, impovement at 3T 5 93%) (Fig. 6). The Effect of Dopping One Site on Between-Site Reliability (Table VII) Figue 6. Effect of dilating ROIs. Between-site ICCs fo median PSC with and without ROI dilation. Note the impoved eliability with dilated ROIs especially at 3T. 3 and 5% of PSC estimates. Fo median PSC (Fig. 5A) smoothness adjustment impoves eliability fo 9 of 12 measues (Wilcoxon Test, P , two-tailed). Fo median CNR (Fig. 5B) smoothness adjustment impoves eliability fo 10 of 12 measues (Wilcoxon Test, P , two-tailed). The notion of dopping one site to incease between-site eliability is obviously a dastic step, but in some cases may be waanted. At the least, an analysis of eliability with and without a site can be a poweful diagnostic tool. In Table VII, the effect of dopping one site at a time fom the vaiance components and between-site eliability calculations is compaed to the case whee all sites ae included. Fo this analysis, all the data come fom the BV ROI exclusively. Dopping SITE 2 fom the analysis inceased the eliability of median PSC fo 1.5T scannes fom 0.44 to Dopping SITE 4 fom the analysis of median CNR fo 1.5T scannes inceased the eliability fom 0.72 to Fo both measues at 3T, SITE 6 site was binging down the eliability. Fo CNR, the change was TABLE VII. The effect of dopping one site at a time on vaiance components and between-site eliability Site that was dopped Measue ROI extaction method Field Site Subject Site 3 subject Visit Residual Between-site ICC ALL_IN CNR Median SITE 1 CNR Median SITE 2 CNR Median SITE 3 CNR Median SITE 4 CNR Median SITE 5 CNR Median ALL_IN PSC Median SITE 1 PSC Median SITE 2 PSC Median SITE 3 PSC Median SITE 4 PSC Median SITE 5 PSC Median ALL_IN CNR Median SITE 6 CNR Median SITE 7 CNR Median SITE 8 CNR Median SITE 9 CNR Median ALL_IN PSC Median SITE 6 PSC Median SITE 7 PSC Median SITE 8 PSC Median SITE 9 PSC Median

11 Fiedman et al. The bette the estimate of the PSC measue o CNR measue, the highe the eliability should be. One way to impove the estimates is to base them on aveages acoss uns. Such an analysis is pesented in Figue 7, fo test etest eliability. Thee ae six ROIs, two field stengths, and two measuement types (PSC and CNR) o 24 total analyses, all of which ae plotted in Figue 7. The six panels wee used to spead the cuves out fo visibility, and the souce data fo individual cuves ae not identified fo the pesent pupose. The points epesent eithe data fom un 1, the aveage of uns 1 and 2, the aveage of uns 1, 2, and 3 o the aveage of all fou uns. The mean cuve on the left plots pedicted values and standad eos fom a polynomial egession (F , df 5 1.7, 40, P ). As pedicted, test etest eliability inceases with aveaging, but not in evey single case. Reliability fo the aveage of fou uns is highe than eliability fo a single un (t , df 5 23, P , paied t-test, onetailed). The aveage of fou uns demonstates highe eliability than the aveage of thee uns in 18 of 24 cases (t , df 5 23, P , paied t-test, one-tailed). Clealy one way to enhance test etest eliability is to incease the numbe of uns, and it seems likely that eliability will continue to incease beyond fou uns. The patten of inceasing eliability with inceasing the numbe of uns included in the aveage was not consistently appaent fo between-site eliability estimates. Concatenating Reliability Enhancing Steps (Fig. 8) Figue 7. Effect of inceasing numbe of uns. Relationship between the numbe of uns contibuting to the aveage estimate (abscissa) and test etest ICC fo 24 measues [two measuement types (PSC vs. CNR), six ROIs, and two field stengths). In the left most panel, the pedicted means and standad eos ae plotted fom a epeated measues polynomial contast model. The 24 cuves ae spead acoss six panels to enhance visibility of each cuve. The souce of the data in each cuve is not identified in the figue. quite maked ( ), wheeas fo PSC the change was modest ( ). The Effect of Numbe of Runs on Test Retest and Between-Site Reliability (Fig. 7) To illustate the effects of accumulating the benefits of seveal steps to impove between-site eliability, we began with the oiginal data fo median PSC fo 3T scannes (Fig. 8). In the fist step, SITE 6 site was dopped and this led to a substantial incease in between-site eliability. In the second step, we dilated the all of the ROIs as descibed above. This futhe inceased between-site eliability. In the final step, we adjusted fo smoothness diffeences between sites. This latte effect was almost unnoticeable except fo the BV ROI. The median initial eliability was 0.26 and the median final eliability was 0.58, a statistically significant (Wilcoxon, P , one-tailed) impovement of 123%. DISCUSSION A key goal of this epot was to assess test etest eliability and between-site eliability fo a multicente fmri study involving 10 sites and a obust sensoimoto activation task. Measues of eliability ae obtained fom a vaiance components analysis of the fmri activations fo the study in which five subjects visited each of 10 sites fo two visits (on consecutive days). In geneal, test etest eliability was high, but initially, between-site eliability was low. Seveal methods wee evaluated fo impoving betweensite eliability. By employing multiple methods, maked inceases in between-site eliability wee noted. Figue 8. Concatenating steps to impove eliability. Effect of a seies of steps, descibed in the text, on between-site eliability fo median PSC fom 3T scannes. 968

12 Test Retest and Between-Site Reliability Test Retest Reliability We epot high test etest eliability fo a 1 day inteval fo activations fom a obust sensoimoto task in seveal ROIs. This indicates that fmri tasks can be eliable when etested on subsequent days within a site, but this eliability level will likely depend highly on the obustness of the task and the numbe of uns studied. The pesent epot is simila in seveal espects to the ecent epot of Aon et al. [2006]. These authos compaed test etest eliability on a leaning task (without pactice effects) with a test etest inteval of 1 yea. The dependent measue used fo ICC calculation in Aon et al. [2006] was a measue of signal change (athe than CNR), and signal changes wee mean estimates fom ROIs. All of the ROI-based test etest ICCs Aon et al. [2006] epoted wee in the excellent ange. Kong et al. [2007] epoted test etest ICCs fo a unilateal finge tapping task, which included pimay moto and supplementay moto aeas. They employed a measue of signal magnitude and obtained an ICC fo the left moto aea of 0.68 quite simila to the ICCs fo the LM and RM ROIs epoted heein (Fig. 3). The test etest eliability fo the supplementay moto aea in the Kong et al. [2007] study was 0.51 somewhat lowe than we epot heein. As expected, an incease in the numbe of uns used in an aveage estimate was associated with incease in the test etest eliability. The effect of inceasing numbes of uns on the estimate of test etest eliability was geatest fo the diffeence between a single un and two uns, and less fo the addition of each addition un, but was still statistically significant fo the diffeence between thee uns and fou uns. Since we an only fou uns of the sensoimoto task, we could not empiically estimate the effect of moe uns. Howeve, the impovement in eliability with inceasing uns should incease as a function of 1/ N uns. Thus, it seems unlikely that significant impovement in eliability with inceasing numbes of uns will continue beyond 6 8 uns. On the othe hand, fatigue is accumulating though the scanning session, and it seems likely that attention and motivation will wane as moe and moe uns ae tested. Thus, we pedict that the empiical elationship between numbe of uns and eliability will peak at some point and then eithe plateau o actually decline. Theefoe, we ecommend that in peliminay studies pio to a majo multicente study, the fmri task be tested on many uns at one o moe sites, to detemine the optimal numbe of uns to enhance eliability. Pevious fmri eliability studies have epoted inceased eliability when moe data ae collected. Fo example, Genovese et al. [1997] epoted inceased test etest eliability within a scanning session as a function of inceasing the un duation (numbe of volumes collected). In a PET CBF study, Gabowski and Damasio [1996] epoted that the eplication ate fo activations inceased makedly when the analysis was based on two PET uns athe than a single un. Of paticula inteest is the epot by Maita et al., [2002], in which the estimates of eliability wee based on 2 12 eplications (1 un pe visit, 12 visits) of a finge-tapping task. They epot that the gain in eliability... is most ponounced when we move fom 2 to 3 eplications, and tapes off substantially at aound 5 o 6 eplications. Between Site Reliability It was hypothesized that a measue of pecent signal change (PSC) would poduce highe between-site eliability than a measue of contast-to-noise-atio (CNR). This hypothesis was based on the notion that noise would likely be highly vaiable acoss sites and this would lowe the eliability of the CNR measue but not the PSC measue. This was appaently not the case, since the compaison of between-site eliability based on PSC vesus CNR did not show a maked advantage fo eithe measue, and actually showed a slight advantage to CNR measues. This statement only applies to the methods of assessment of PSC and CNR tested heein. Thee ae numeous methods fo computing PSC and CNR that may poduce diffeent esults. Fo example, instead of PSC and CNR measues fom an FIR-deconvolution, one could have used PSC and CNR measues fom the egession of a pedetemined tempoal model [events convolved with a canonical hemodynamic esponse function (HRF)]. Futue eseach will be equied fo a compehensive compaison of diffeent measues of PSC and CNR. Cohen and Dubois [1999] compaed the stability of egession b-weights (a signal magnitude measue) to the stability of numbe of activated voxels and found the fome to be much moe stable than the latte. Howeve, high ICC values have been epoted fo t-values (a CNR measue) [Specht et al., 2003]. These authos employed an event-elated visual activation paadigm on two visits and computed within-site ICC estimates fo each voxel based on t-values. In thei attend condition, thee was vey high eliability fo t-values in the pimay visual cotex (ICCs > 8). We also hypothesized that the extaction of a median IRF fom each ROI would lead to moe eliable esults than the extaction of the maximum IRF. In many cases, data based on the maximum IRF was moe eliable than data based on the median IRF, and no clea winne was obvious. We also hypothesized that contolling fo site diffeences in the native smoothness of the images [Fiedman et al., 2006] would impove between-site eliability. This was based on the notion that CNR measues would be elated to image smoothness, and that scannes that poduced smoothe images would show highe CNR. This tuned out to be the case, since adjusting fo smoothness diffeences in CNR pio to between-site eliability estimation poduced a statistically significant incease in eliability. Futhemoe, adjusting fo smoothness diffeences in PSC also poduces a statistically significant incease in between-site eliability. The slope of the elationship between image smoothness and PSC was negative, indicating that the 969

13 Fiedman et al. smoothe the data, the lowe the PSC estimate. This could be the esult of spatial smoothness tending to ound off the peaks and toughs of the aw image data and thus leading to a educed PSC. Ou hypothesis that inceasing the ROI size would incease between-site eliability was bone out, paticulaly at 3T. Pehaps this is simply due to the fact that ou oiginal ROIs wee too small, even though they wee defined to include activations fom all sites. The maked beneficial effect of ROI dilation at 3T could be due to the established fact that spatial image distotions ae inceased at 3T compaed with 1.5T. In the pesent study, no B0-distotion coection pocedues wee applied fo the EPI acquisitions. Such coection pocedues should educe vaiance in the activation sites, especially at 3T. Futhemoe, diffeent spatial nomalization techniques may have diffeential effects on eliability. Futue FBIRN data acquisitions employ B0- coection pocedues, so this souce of unwanted vaiance should be minimized somewhat going fowad. The ROIs we studied eflected common activation acoss all scannes. Ou study examined souces of vaiation in signal magnitude fo voxels whee the P-value was consistently above a statistical theshold acoss sites. We chose this definition of an ROI because conjunction methods of deiving ROIs ae commonly used in fmri studies [Fiston et al., 1999; Quintana et al., 2003]. This ROI definition, based on multisite consistency, should poduce an elevated between-site eliability compaed to ROI definitions that ae not based on multisite consistency. Howeve, employment of this ROI without othe eliability enhancing pocedues poduced low eliability estimates (1.5T median: 0.22, 3T median: 0.25). Employment of ROIs that ae not defined in efeence to multisite activation consistency (e.g., atlas-based ROIs) ae likely to poduce eliability estimates even close to 0.0. The pesent study emphasizes the citical impotance of ROI definition on eliability. Clealy, follow-up studies which compae diffeent methods of ROI definition ae waanted. We also examined the notion that inceasing the numbe of uns of a task would incease the between-site eliability. Although such an effect was unequivocally demonstated fo test etest eliability, the effect of inceasing the numbe of uns on between-site eliability was not consistently obseved. Between-site eliabilities wee geneally poo initially, and the lack of effect of inceasing the numbe of uns may simply eflect the notion that uneliable entities ae not made moe eliable by epeated sampling. The notion of combining uns to incease accuacy only makes sense if thee ae no pactice effects fo the task unde consideation. A numbe of fmri studies have now documented pactice effects fo some tasks [fo eview, see Kelly and Gaavan, 2005]. If one is to gain the eliability incease associated with aveaging uns of a task, one should establish that pactice effects ae not an impotant chaacteistic of the task. The notion of dopping a site to impove between-site eliability is a dastic step. Nonetheless, the maked impovement in between-site eliability fo PSC at 3T afte dopping the SITE 6 site suggests that in the pesent case, this might be waanted. Regadless of whethe one chooses to actually dop a site, the dop-one-site analysis is a useful tool to highlight sites that ae paticulaly influential in inceasing o deceasing between-site eliability. In the case of SITE 6, we have shown in pevious publications [Fiedman and Glove, 2006; Fiedman et al., 2006] that this site had the weakest activations of any of the highfield scannes. Since we ae employing a eliability estimate that assesses absolute ageement, any site diffeence in PSC o CNR will lowe eliability. We have ecently been infomed that this site has been seveely impacted by envionmental noise fom a neaby subway tain fo yeas. Dopping SITE 2 fom the 1.5T scannes was also associated with a substantial incease in between-site eliability. As we have pointed out in an ealie epot [Fiedman et al., 2006], SITE 2 site employed a athe sevee k-space (apodization) filte and theefoe poduced unusually smooth images. This maked incease in smoothness would be expected to poduce makedly elevated CNR estimates. Removal of this site would homogenize CNR acoss sites and lead to geate between-site eliability. Removal of SITE 4 was associated with inceased eliability fo PSC at 1.5T. Futhe examination evealed that SITE 4 had the lowest PSC estimates of any 1.5T site fo fou of five ROIs. This was an olde Picke 1.5T scanne that has since been decommissioned and eplaced. This analysis emphasizes the impotance of fully evaluating the pefomance of each scanne pio to inclusion in multicente studies. A single unusual scanne can makedly affect between-site eliability estimates. Anothe goal of this epot was to illustate the usefulness of using within-site ICC estimates as a benchmak fo between-site ICC estimates. In the multicente context, between-site eliability has only subject vaiance in the numeato, wheeas test etest eliability has subject, site, and subject-by-site vaiance in the numeato. So, by definition, test etest eliability will always be highe than between-site eliability. Howeve, test etest eliability can be assessed in one o seveal unicente studies. If one accepts the goal that between-site eliability should be in the good ( ) o excellent ange (above 0.75), it is pobably wise to assess test etest eliability at one o seveal sites pio to initiation of a multicente study. If test etest eliability is not in the excellent ange, it seems unlikely that between-site eliability will be in the good ange, due to the additional vaiance due to site and the subject-by-site inteaction. It seems likely that thee will always be some degadation of eliability due to changes in hadwae and setting. A concen fo the pesent study and fo futue studies of within-site o between-site eliability is the sample size of the eliability study. Seveal appoaches to pospectively estimating sample sizes fo eliability studies have been poposed [Chate, 1999; Giaudeau and May, 2001; Walte et al., 1998]. These appoaches elate sample sizes to the widths of the confidence intevals aound ICC esti- 970