The following page contains the final YODA Project review approving this proposal.

Transcription

1 The YODA Project Research Proposal Review The following page contains the final YODA Project review approving this proposal. The Yale University Open Data Access (YODA) Project Yale University Center for Outcomes Research and Evaluation (CORE) 1 Church Street, Suite 200, New Haven, CT 06510

2 The YODA Project External Research Proposal Review - Final (Protocol #: ) Reviewers: Nihar Desai Ezekiel Emanuel Cary Gross Harlan Krumholz Richard Lehman James Neaton Bruce Psaty Rita Redberg Review Questions: 1. Is the scientific purpose of the research proposal clearly described? Joseph Ross Dennis Thompson Deborah Zarin Decision: 2. Will proposed research create or materially enhance generalizable scientific and/or medical knowledge to inform science and public health? 3. Can the proposed research be reasonably addressed using the requested data?, or it's highly likely 4. Does the proposed research provide strong rationale for why the requested data can uniquely be used to address the proposed project aims? 5. Does proposed research provide clear justification for access outside the secure data sharing platform? N/A 6. Does proposed research describe the protections in N/A place to ensure data security outside of the platform? 7. Recommendation for this data request: Comments: Approve The data request has been revised and the software flexmirt will be added to the secure data access platform, where the investigator will work only with Janssen data. The Yale University Open Data Access (YODA) Project Yale University Center for Outcomes Research and Evaluation (CORE) 1 Church Street, Suite 200, New Haven, CT 06510

3 The YODA Project Research Proposal Review Revisions were requested during review of this proposal. The following pages contain the original YODA Project review and the original submitted proposal. The Yale University Open Data Access (YODA) Project Yale University Center for Outcomes Research and Evaluation (CORE) 1 Church Street, Suite 200, New Haven, CT 06510

4 The YODA Project External Research Proposal Review - Revisions Requested (Protocol #: ) Reviewers: Nihar Desai Ezekiel Emanuel Cary Gross Harlan Krumholz Richard Lehman James Neaton Bruce Psaty Rita Redberg Review Questions: 1. Is the scientific purpose of the research proposal clearly described? Joseph Ross Dennis Thompson Deborah Zarin Decision: 2. Will proposed research create or materially enhance generalizable scientific and/or medical knowledge to inform science and public health? 3. Can the proposed research be reasonably addressed using the requested data?, or it's highly likely 4. Does the proposed research provide strong rationale for why the requested data can uniquely be used to address the proposed project aims? 5. Does proposed research provide clear justification for access outside the secure data sharing platform? No 6. Does proposed research describe the protections in No place to ensure data security outside of the platform? 7. Recommendation for this data request: Comments: Not Approve The request does not specifically address 2 of the points identified for data outside the secure platform -justification for request and security. The reasons appear to be able to combine the data with other data sources and to use the flexmirt software instead of SAS. I am not familiar with item analysis so cannot judge how essential the flexmirt software is for their research. SAS does have item analysis routines. An alternative to using obtaining the data and pooling it with the other data they mention would be to compute summary statistics in SAS and pool the summary statistics. I am sympathetic to their request. It is easier to use software you are familiar with and also easier to do that with the raw data from trial they propose to use. Their proposal seems well thought out. Thus, I would grant their request if they address the security of the data outside of the platform. It is unclear why the data are going to be retained on a USB, rather than on a server or desktop computer (that has the appropriate security in place). There is no clear description of the processes to ensure data security outside the platform. The statistical analysis plan mentions that models will most likely be estimated in flexmirt. R can also be used for IRT models, and R is available within the SAS platform. In an effort to find a way to make this data sharing possible, would the investigator consider using R for analysis instead? The goal is to combine the Janssen data with data from Clinical Study Data Request and UCLA. Even if the software could be added to the platform, the other data would need to be added as well in order to combine datasets. Please confirm whether these other data (per the agreements you have in place for them) can be uploaded to the data sharing platform. Alternatively, summary analyses from the Janssen data can be exported and then combined with the other data outside of the platform, if that is suitable. The Yale University Open Data Access (YODA) Project Yale University Center for Outcomes Research and Evaluation (CORE) 1 Church Street, Suite 200, New Haven, CT 06510

5 Principal Investigator First Name: Alex Last Name: Sturm Degree: Ph.D. Primary Affiliation: University of California, Los Angeles Phone number: Address: 300 Medical Plaza Suite 1534 City: Los Angeles State or Province: California Zip or Postal Code: Country: United States General Information Key Personnel (in addition to PI): Last name: McCracken Degree: MD Primary Affiliation: UCLA First Name: James First Name: Li Last name: Cai Degree: PhD Primary Affiliation: UCLA Are external grants or funds being used to support this research?: No external grants or funds are being used to support this research. yoda_coi_mccracken.pdf yoda_project_coi_form_for_data_requestors_2016_-_li-signed.pdf yoda_project_coi_form_for_data_requestors_2016_-_alex.pdf Certification Certification: All information is complete; I (PI) am responsible for the research; data will not be used to support litigious/commercial aims. Data Use Agreement Training: As the Principal Investigator of this study, I certify that I have completed the YODA Project Data Use Agreement Training Associated Trial(s): NCT A Multi-center Randomized Parallel Group Study Evaluating Treatment Outcomes of Concerta (Extended Release Methylphenidate) and Strattera (Atomoxetine) in Children With Attentiondeficit/Hyperactivity Disorder What type of data are you looking for?: Individual Participant-Level Data, which includes Full CSR and all Page 1 of 6

6 supporting documentation Research Proposal Project Title Applications of Item Response Theory to Clinical ADHD Research: Analysis of the ADHDRS-IV and Increased Precision of Treatment Effect Estimation Narrative Summary: Rating scales serve as a necessary tool for clinicians and therefore accurate measurement of symptomatology is essential. Item response theory (IRT), a latent variable model, presents an opportunity to improve the way we measure baseline clinical symptoms of ADHD and estimate change over the course of treatment. This study aims to use IRT to evaluate the psychometric properties of the ADHD-RS-IV in a large, combined baseline sample of children and adolescents. Study 2 will model change in ADHD symptoms as a function of treatment within the IRT framework in order to examine differences in treatment sensitivity as measured by latent traits of ADHD. Scientific Abstract: Background: Item response theory (IRT), a latent variable model, presents an opportunity to improve the way we measure and evaluate clinical symptoms of ADHD in addition to treatment response. Objective: This study aims to (1) evaluate the psychometric properties of the ADHDRS-IV and to determine differences by age, gender, and comorbidity in a large, combined baseline sample of youth using IRT and (2) model change in ADHD symptoms as a function of treatment within the IRT framework using a multidimensional model of ADHD symptoms. Study Design: Study 1 will employ a cross-sectional, pooled, multi-level analysis of baseline ADHD-RS-IV items. Study 2 will use longitudinal ADHD-RS-IV from the requested trial to compare latent dimension treatment sensitivity through modeling of latent change in the ADHD trait within a multi-level IRT framework. Participants: Participants include all children ages 6 to 17 who were rated using the ADHD-RS-IV by a trained clinician. Main Outcome: IRT item parameter estimates of baseline ADHD-RS-IV items and latent dimension means and variances to evaluate change from baseline to endpoint. Measure(s): ADHD-RS-IV, age, gender, and comorbid diagnose. Statistical Analysis: Item response theory will be used to model the psychometric properties of the ADHDRS-IV. A multilevel item factor model will allow the investigator to model the nested nature of the data. A multiple group multilevel longitudinal item bifactor model will be used to estimate latent change scores in the ADHD construct over treatment. Brief Project Background and Statement of Project Significance: Attention-deficit/hyperactivity disorder (ADHD) is an externalizing behavioral disorder that affects between 5 and 10 percent of youth (Polanczyk, Silva de Lima, Biederman, & Rohde, 2007). A number of rating scales exist for the purpose of measuring symptoms of ADHD for children and adolescence including the CPRS (Conners, Sitarenios, Parker, & Epstein, 1998), SNAP-IV (Swanson, 1992), and the ADHDRS-IV (DuPaul, Anastopoulos, Power, Reid, Ikeda, & McGoey, 1998). Rating scales serve as a necessary tool for clinicians who wish to determine presence and severity of ADHD symptoms, in addition to monitoring response to treatment (Conners, 1997). Because the best method for detecting the effectiveness of a treatment is through perceived changes in symptoms, accurate measurement of symptomatology is essential. However, as advances in quantitative methodology have improved measurement in domains such as physical functioning, emotional distress, and pain, clinical research in ADHD has lagged behind. Item response theory (IRT), a latent variable model, presents an opportunity to improve the way we measure baseline clinical symptoms of ADHD and treatment response. Successful IRT modeling of rating scales has enabled the Patient-Reported Outcomes Measurement Information System (PROMIS) Cooperative Group and NIH to create more precise, flexible, and reliable forms. Traditionally, change in ADHD symptoms over the course of treatment are estimated using summed scores. However, using this approach it appears that all pharmacotherapies produce relatively uniform reductions across inattention and hyperactivity/impulsivity with varying effect sizes depending on the specific treatment. Unlike summed scores, latent variable models produce mutually orthogonal dimensions. It is possible that when independent sources of variability are identified, we may find that different treatments in fact produce differential reduction across latent dimensions. To maximize the potential of IRT Page 2 of 6

7 application in ADHD measurement, we must also capitalize on the increasing availability of data previously collected by investigators and companies that have a stake in the advancement of research and treatment. However, latent variable models such as IRT with many model parameters require big samples for stable and precise estimation. Now, many samples exist and can be combined to both improve our measurement and understanding of symptoms and to identify targeted treatment approaches. This proposal will be presented in two parts. Study 1 will use IRT to evaluate the psychometric properties of the ADHDRS-IV in a large, combined baseline sample of children and adolescents. Study 2 will model change in ADHD symptoms as a function of treatment within the IRT framework in order to examine item-level treatment sensitivity across several pharmacotherapies used to treat ADHD. The results of this IRT analysis are predicted to identify a more concise and presumably more sensitive change index which could have research and clinical value both in diagnosis and in treatment assessment. Results will be submitted for publication in psychological methods journals. Specific Aims of the Project: Psychometric properties of the ADHDRS-IV collected at intake will be evaluated using IRT. More specifically, this study aims to (1) use IRT to confirm the pre-treatment multidimensional structure of the 18 ADHDRS-IV items in children and adolescents (2) evaluate the item parameters of the IRT model including discrimination, thresholds, and the unique information provided by items at intake to determine the unique contribution of each item to the construct of ADHD as a whole (3) determine if a short form for ADHD may be developed through closer examination of IRT item parameter estimates and (4) determine if gender, age or comorbid diagnoses affect IRT item parameters, which would inform our understanding of ADHD. Study 2 will use the best fitting model of ADHD-RS-IV items to model change in latent dimensions of ADHD over the course of treatment. We hypothesize that different pharmacotherapeutic approaches will produce differential reduction in orthogonal latent dimensions of ADHD symptoms. What is the purpose of the analysis being proposed? Please select all that apply. Research that confirms or validates previously conducted research on treatment effectiveness Participant-level data meta-analysis Participant-level data meta-analysis will pool data from YODA Project with other additional data sources Research Methods Data Source and Inclusion/Exclusion Criteria to be used to define the patient sample for your study: A baseline sample of children and adolescents who were rated using the ADHD-RS-IV are needed to estimate the IRT model. It is important that this sample reflects the population of interest, in this case children and adolescents who may possess symptoms of ADHD. For this reason, baseline demographic and ADHD-RS-IV data is requested from Janssen study NCT Baseline samples have been received from Clinical Study Data Request and investigator-initiated trials at UCLA. A very large sample is required to produce stable model parameter estimates in IRT for the ADHD-RS-IV and therefore these baseline samples will be combined for analysis. Screening or baseline item-level ADHD-RS-IV data for all 6 to 17 year-old study participants will be used. There are no other inclusion/exclusion criteria for participants to be included in the analysis. In addition, baseline and treatment endpoint ADHD-RS-IV scores will be used from the requested trial to estimate differences in latent change in symptoms of ADHD over the course of treatment. Main Outcome Measure and how it will be categorized/defined for your study: In study 1, psychometric properties of the ADHDRS-IV collected at baseline will be evaluated using IRT. More specifically, this study aims to (1) use IRT to confirm the pre-treatment multidimensional structure of the 18 ADHDRS-IV items in children and adolescents (2) evaluate the item parameters of the IRT model including discrimination, thresholds, and the unique information provided by items at baseline to determine the utility of each item and (3) generate IRT item parameter estimates that will be used in the analyses of treatment effect where small sample sizes limit the ability to estimate both item parameters, and change over time in the IRT framework. In study 2, latent change in ADHDRS-IV inattention and hyperactivity/impulsivity subscales will be evaluated from baseline to last available treatment end-point to examine treatment differences. Page 3 of 6

8 Main Predictor/Independent Variable and how it will be categorized/defined for your study: In the primary model, there will be no independent variables as the purpose is to fit the general IRT model. However, during sub-group analysis, age, gender, and comorbid diagnoses will serve as grouping variables in separate models. Children ages 7 to 10 will be grouped and children ages 11 to 17 will be grouped. A decision regarding how to group by age was made based on literature presented regarding the statistically significant increase in inattention symptoms for children over the age of 11 (Bussing et al., 2008) and the general decrease in symptoms of hyperactivity/impulsivity as children age (Lahey, Pelham, Loney, Lee, & Willcutt, 2005), in the late prepubertal period of development. Gender will be evaluated as any differences in item parameter estimates between males and females. In the latent change treatment model, baseline and treatment endpoint ADHD-RS-IV scores will be used. Other Variables of Interest that will be used in your analysis and how they will be categorized/defined for your study: The requested study was a multi-site trials and data was collected across the U.S. The varying prevalence and treatment rates across the U.S. (CDC, 2010) support the need to model the baseline data within a multilevel framework to account for site differences of ADHD-RS-IV baseline ratings. A multilevel item factor model will allow the investigator to model the nested nature of the data, where study participants (level-1 units) were nested within sites (level-2 units). Therefore, site will be used as another variable of interest. Statistical Analysis Plan: Item response theory will be used to model the psychometric properties of the ADHDRS-IV including item discrimination, thresholds, and the unique information about the latent trait of ADHD provided by the 18 items. Specifically, a graded response, multidimensional model. A multilevel item factor model will allow the investigator to model the nested nature of the data. Sub-group differences across gender, age, and comorbid diagnoses will be explored using differential item functioning using a multi-step Wald chi-square procedure (Woods, Cai, & Wang, 2013; Langer, 2008; Cai et al., 2011)). To control the inflated error rate caused by multiple DIF comparisons, the false discovery rate procedure (Benjamini & Hochberg, 1995) will be used. A longitudinal, multidimensional IRT model will be used to estimate change from baseline to treatment endpoint to compare pharmacotherapies used in the requested trial. Item parameters will be fixed at values determined in study 1. Models will likely be estimated in flexmirt (Cai, 2013) using maximum marginal likelihood via the Bock-Aitkin EM algorithm (Bock & Aitkin, 1981), which will also handle missing data. The use of flexmirt is necessary due to the complexity of the IRT model (e.g. multiple group, multilevel). Several indices of model fit will be reported and used to assess the fit of the model. Due the possibility of many response patterns (4^9) and a sparse contingency table, the M2 statistic will be reported (Cai & Hansen, 2013). Root mean square error of approximation (RMSEA; Steiger & Lind, 1980) will also be reported in addition to marginal fit (chi-square) and LD (chi-square) statistics for further evaluation and discussion of model fit. Project Timeline: The project will start as soon as data, or decisions regarding data, have been received. Analyses will be completed within three months following receipt of data. The manuscript will be drafted by the end of the 2016 calendar year and immediately submitted for publication. Dissemination Plan: The purpose of these analyses is primarily for an extension of dissertation research. The plan is to complete all analyses within 2 months of data receipt and submit for publication 2 months after completion of analyses. The manuscript will be submitted to a psychology/psychiatry journals such as Journal of Child Psychology and Psychiatry, or Journal of Attention Disorders. Bibliography: Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), Page 4 of 6

9 Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, Bussing, R., Fernandez, M., Harwood, M., Hou, W., Garvan, C. W., Eyberg, S. M., & Swanson, J. M. (2008). Parent and teacher SNAP-IV ratings of attention deficit hyperactivity disorder symptoms psychometric properties and normative ratings from a school district sample. Assessment, 15(3), Cai, L. (2013). flexmirt R version 2: Flexible multilevel multidimensional item analysis and test scoring [Computer software]. Chapel Hill, NC: Vector Psychometric Group. Cai, L., & Hansen, M. (2013). Limited information goodness of fit testing of hierarchical item factor models. British Journal of Mathematical and Statistical Psychology, 66(2), Cai, L., Thissen, D., & du Toit, S. H. C. (2011). IRTPRO: Flexible, multidimensional, multiple categorical IRT modeling [Computer software]. Lincolnwood, IL: Scientific Software International. Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S.,... & Group, P. C. (2010). The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: Journal of clinical epidemiology, 63(11), Centers for Disease Control and Prevention. (2015). Attention-deficit/hyperactivity disorder (ADHD). Retrieved from Conners, C. K. (1997). Rating scales in attention-deficit/hyperactivity disorder: use in assessment and treatment monitoring. The Journal of clinical psychiatry, 59(Suppl 7), Conners, C. K., Sitarenious, G., Parker, J. D., & Epstein, J. N. (1998). The revised Conners Parent Rating Scale (CPRS-R): factor structure, reliability, and criterion validity. Journal of Abnormal Child Psychology, 26(4), DuPaul, G. J., Anastopoulos, A. D., Power, T. J., Reid, R., Ikeda, M. J., & McGoey, K. E. (1998). Parent ratings of Attention-Deficit/Hyperactivity Disorder symptoms: Factor structure and normative data. Journal of Psychopathology and Behavioral Assessment, 20(1), Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates Inc.; Mahwah, NJ. Lahey, B. B., Pelham, W. E., Loney, J., Lee, S. S., & Willcutt, E. (2005). Instability of the DSM-IV subtypes of ADHD from preschool through elementary school. Archives of General Psychiatry, 62(8), Langer, M. (2008). A reexamination of Lord s Wald test for differential item functioning using item response theory and modern error estimation (Unpublished doctoral dissertation). Polanczyk, G., Silva de Lima, M., Horta B. L., Biederman, J., & Rohde, L. A. (2007). The worldwide prevalence of ADHD: a systemic review and metaregression analysis. American Journal of Psychiatry, 164(4), Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometric Monograph, No. 17. Steiger, J. H. & Lind, J. C. (1980). Statistically Based Tests for the Number of Common Factors. Paper presented at the annual meeting of the Psychometric Society, May, Iowa City, IA. Woods, C. M., Cai, L., & Wang, M. (2013). The Langer-Improved Wald test for DIF testing with multiple groups: evaluation and comparison to two-group IRT. Educational and Psychological Measurement, 73(3), Supplementary Material: yoda_project_research_proposal_direct_data_access_may_2016.pdf Page 5 of 6

10 Page 6 of 6