OBSERVER-REPORTED OUTCOMES IN CLINICAL TRIALS: CHALLENGES AND SOLUTIONS Donald L. Patrick, PhD, MSPH, Professor, University of Washington, Seattle, WA, USA Rob Arbuckle, MSc, Director, Endpoint Development & Outcomes Assessment, Adelphi Values, Bollington, United Kingdom; Laurie Beth Burke, MPH, Founder, LORA Group, LLC, Adjunct Professor, University of Maryland, Royal Oak, MD, USA WORKSHOPS - SESSION III Tuesday, 11 November 2014 16:45-17:45 Disclaimers The views and opinions expressed here are those of presenters and should not be attributed to ISPOR, FDA/EMA, or the University of Washington These PowerPoint slides are the intellectual property of the individual presenters and are protected under the copyright laws of the United States of America and other countries. Used by permission. All rights reserved. All trademarks are the property of their respective owners. 1
Types of Informant-Reported Outcomes Objectives: you will be able to Discuss rationale and conceptual basis for selecting ObsRO assessments for clinical trials conducted during medical product development Identify the methodological and practical challenges in developing, implementing, and interpreting ObsROs, either existing or new 2
Outline ObsROs in Regulatory Context - 10 minutes Laurie Burke An ObsRO for Children with Cystic Fibrosis Ages 0-6 Years - 15 minutes Donald Patrick An ObsRO for Restless Legs Syndrome (RLS) in Children Ages 6-11 Years - 15 minutes Rob Arbuckle Discussion 20 minutes Clinical Outcome Assessments Biomarkers Performance Clinician-Reported Observer-Reported Signs Patient-Reported Cholesterol (coronary disease) C-reactive protein (inflammatio n) Motor (timed 25 foot walk test) Sensory (visual acuity, test reading) Cognition (memory recall, or other cognitive testing (e.g., digit symbol substitution test). Global impression of severity/change Algorithmic ratings Radiographic readings with human interpretation Cough Activity level Sleep Symptoms Function Feelings Perceptions Survival 3
Development, Validation, and Use of Observer-Reported Outcomes in Clinical Trials: Regulatory Perspective Laurie Burke LORA Group, LLC, Royal Oak, MD USA ISPOR 17 th Annual European Congress 11 November 2014 Amsterdam 7 Presentation Outline Regulatory view of outcome assessment to support a conclusion of treatment benefit Path to treatment benefit measurement Roadmap to Patient-Focused Outcome Measurement Specific regulatory concerns for observerreported measurement of treatment benefit 4
Regulatory Evolution of Treatment Benefit 1962 to Present: Added Focus on Efficacy 1962 Amendments to FFDCA NDA required to prove comparative effectiveness (placebo or active comparator) Randomization, blinding, statistical significance, replication 2009 Final FDA PRO Guidance 2014 Final FDA Drug Development Tool Qualification Guidance Biomarkers Clinical outcome assessments Animal models 2014 Formal EMA/FDA parallel qualification review and collaboration Trajectory toward better information for all decisionmakers 9 US Regulatory Push for Pediatric Studies in Drug Development Pediatric Research Equity Act of 2007 (PREA) Best Pharmaceuticals for Children Act of 2007 (BPCA) FDA Safety and Innovation Act of 2012 (FDASIA) Makes BPCA and PREA permanent Requires earlier submission of a pediatric study plan (EOP2) Emphasizes need for suitable measurement tools to evaluate treatment benefit Survival Symptoms Daily functioning Mandates patient participation in medical product discussions PDUFA Reauthorization for FY 2013-17 Additional FDA clinical and statistical staff with PRO expertise Patient-focused drug development initiative to incorporate the patient perspective into FDA decision-making 20 PFDD public meetings on specific diseases 10 5
Recent US Regulatory Events Recognize the Importance of the Patient Voice House Energy and Commerce Health Panel hearing on possible regulation of patientreported outcomes, 11 July 2014 Woodcock said FDA needs stakeholders' help in developing the science to measure outcomes not regulation FDA/Brookings: Enhancing the Development and Use of Patient-Reported Outcomes in Drug Development, July and Oct 2014 Additional guidance for stakeholders on acceptable endpoint measures Additional training for FDA staff New Regulatory/HTA Guidance Focuses on How Patients Feel, Function and Survive FDA Guidance for Industry: Expedited Programs for Serious Conditions Drugs and Biologic, May 2014 A clinical endpoint is a characteristic or variable that directly measures a therapeutic effect of a drug an effect on how a patient feels (e.g., symptom relief), functions (e.g., improved mobility), or survives. All other endpoints are either surrogate or intermediate Terminology Adopted by EUnetHTA, November 2013 A clinical endpoint is a patient relevant endpoint A valid measure of clinical benefit due to treatment The impact of treatment on how a patient feels, functions and survives 12 6
Roadmap to Patient-Focused Outcome Measurement in Clinical Trials Developed by FDA to help drug and instrument developers summarize the information needed in a briefing document Particularly useful in diseases/conditions that are not yet adequately conceptualized in terms of how patients feel and function Developed to avoid the following problems: The measure is not targeted to the clinical trial population The measure does not capture the concept(s) of interest The measure is not sensitive to meaningful change The measure does not detect treatment benefit Clinical trial results do not provide optimal information about patient outcomes 13 Template Cover Template Date Epstein Health, LLC 50 Tice Boulevard,Suite 340 Woodcliff Lake, NJ 07677 7
Identify the Type of Measure Needed to Measure the Concept of Interest? The Choices Are. Survival Clinical outcome assessments (COAs) Influenced by human choices, judgment, or motivation; depend on the cooperation, judgment, motivation, and reporting of observations Performance outcomes (PerfOs) Clinician reported outcomes (ClinROs) Observer reported outcomes (ObsROs) Patient reported outcomes (PROs) Biomarkers A physiologic, pathologic, or anatomic characteristic that is objectively measured and evaluated as an indicator of some normal or abnormal biologic function, process or response to a therapeutic intervention See Guidance for Industry and FDA Staff: Qualification Process for Drug Development Tools, www.fda.gov/cder/guidance, January 7, 2014. 15 Clinical Outcome Assessments Biomarkers Performance Clinician-Reported Observer-Reported Signs Patient-Reported Cholesterol (coronary disease) C-reactive protein (inflammatio n) Motor (timed 25 foot walk test) Sensory (visual acuity, test reading) Cognition (memory recall, or other cognitive testing (e.g., digit symbol substitution test). Global impression of severity/change Algorithmic ratings Radiographic readings with human interpretation Cough Activity level Sleep Symptoms Function Feelings Perceptions Survival 8
Observer-Reported Outcome Measurement: Key Principles Does the score measure something that is meaningful to treatment benefit in the study population? What exactly does the score represent? How is the score related to patient survival, symptoms, or daily functioning? Is there empiric evidence in the targeted population that the measure is sufficiently targeted and comprehensive? Is the scored concept consistent with trial objectives? Is the score reliable? Does not change in stable patients in the targeted population? Minimized variability in the study population Is the measure properly positioned for analysis in the primary/secondary endpoint hierarchy? The number of endpoints possible to test in one study is not limitless Other outcomes that are critical to the interpretation of treatment benefit need to be properly placed in the hierarchy 17 An ObsRO for Children with Cystic Fibrosis Ages 0-6 Years Donald L. Patrick, Todd Edwards, Margaret Rosenthal ISPOR 17 th Annual European Congress 11 November 2014 Amsterdam Acknowledgments: Funding: University of Washington - Institute of Translational Health Sciences, the National Heart, Lung and Blood Institute (grant #1R41HL098985-01A1), and Vertex Pharmaceuticals. 18 9
Background Trials of nutritional and pulmonary therapies needed to improve long-term outcomes in infants and young children Current available endpoints are primarily surrogates: growth, lung function, and respiratory microbiology No observer-reported outcomes (ObsROs) exist for children too young to self-report ObsROs needed to assess treatment benefit When Child-Report Is Not Possible: Verifiable Observation A sign or impact must be able to be reliably detected by a sense or senses: Seen (vision) Heard (auditory) Smelled (olfactory), or Felt (touch) 20 10
Study Aim Develop a content valid CF-specific respiratory, gastrointestinal, and systemic sign ObsRO for use as a clinical trial endpoint in children with CF 6 years of age Methods Semi-structured interviews with parents of children ages 0-6 years diagnosed with CF 76% interviewed within 30 days of contact with the CF clinic for an acute change in child s respiratory status Item content developed by team of clinicians and health service researchers Review of item content by parents locally and clinicians locally and nationally also conducted 11
Interview Sample Results 12
Observable Signs of Respiratory Exacerbation in Cystic Fibrosis (ages 0-6 years) Labored breathing (hear): Even as he s breathing at night, it ll sound like the breathing is more labored, it s harder for him to breathe, and it ll have that kind of crackle-pop kind of thing going on, as well. (#13) Fussiness (see): Well, about a week and a half ago or so, he started just getting fussy, and he wouldn t finish all of his bottle which is not normal for him. And so his appetite was tapering off. He was getting fussier. (#2) Source: UW Cystic Fibrosis ObsRO Study, PI: Rosenfeld, M. 25 Diary Content 13
27 14
Reliability of ObsROs Inter-rater reliability is more appropriate than testretest, although difficult to implement Test-retest is affected by consistency and frequency of observed phenomenon over time, as well as by measurement error KEY POINT -- Reliability not sufficient by itself: How do we know we are measuring the right thing(s) for development, condition, and treatment context? 29 Conclusions Parents able to report on observable signs in relation to respiratory and gastrointestinal functioning Working with clinical partners, identified how parents describe CF signs in relation to diagnostic criteria This ObsRO development fills an important unmet need for at-home sign reporting for this age group Also helps complete instrument development across the life span of CF patients 15
Pediatric Restless Legs Syndrome Severity Scale Parent Questionnaire Arbuckle R, Abetz L, Durmer JS, Ivanenko A, Owens JA, Croenlein J, Bolton K, Moore A, Allen RP, Walters AS, Picchietti DL ISPOR 17 th Annual European Congress 11 November 2014 Amsterdam Funding and Copyright owned by Boehringer Ingelheim Pharma Thanks to all of the clinicians, children and parents who participated in the development of this instrument Heart, Lung and Blood Institute (grant #1R41HL098985-01A1), and Vertex Pharmaceuticals. 31 Background: Restless Legs Syndrome Characterised by a strong urge to move the legs, usually when sitting or lying down, accompanies by an uncomfortable creepy-crawly sensation Results in sleep disturbance and can impact on concentration Significant comorbidity with ADHD (12-25% of P-RLS cases) Observer-reported outcome (ObsRO) required to support secondary endpoint(s) in clinical trials for children aged 6-17 A child PRO was also developed in parallel 16
Objectives To conduct qualitative interviews with children with RLS and parents of children with RLS aged 6-17 to support the development of both a child PRO and a parent ObsRO that might support clinical trial endpoints Child PRO measure of P-RLS symptoms (potential primary endpoint) and functional impacts developed to support clinical trial endpoints Parent ObsRO measure of P-RLS signs and functional impacts developed to supplement child PRO as secondary or exploratory endpoint Focus today is on the ObsRO Hypothesized Endpoint Hierarchy Instrument Concept Endpoint Hierarchy Child PRO P-RLS Symptoms Functional Impacts Primary Endpoint Secondary Endpoint Parent ObsRO Signs of P-RLS Functional Impacts Impact on Parent Secondary Endpoint Secondary/ Exploratory Endpoint Exploratory Endpoint 17
Concept Elicitation Qualitative Analysis Item Development Cognitive Debriefing Methods Semi-structured interviews with 33 children/adolescents aged 6-17 years and 33 parent/caregivers Diagnosis confirmed by expert clinicians Open-ended, non-leading questions to elicit spontaneous data followed by focussed questioning Thematic analysis using Atlas ti software to identify measurement concepts, group quotes by domain Qualitative data and input from expert clinicians used to support development of conceptual framework, instructions, items response scales and recall period 21 children/adolescents and 15 parents returned to participate in cognitive interviews Understanding, interpretation, relevance of concepts tested with parents and children Revisions considered, again with input from the pediatric RLS experts Input from an advisory board of expert clinicians provided expert clinical input throughout 4 pediatric sleep disorder specialists 2 RLS experts who were developers of the adult IRLS Severity Scale Samples for Concept Elicitation and Cognitive Debriefing Diagnoses Age 6-8 years Age 9-11 years Age 12-17years Total Male Female Male Female Male Female Concept Elicitation RLS without ADHD 5 1 3 3 3 3 18 RLS + ADHD 2 2 3 2 3 3 15 Total 10 11 12 33 Cognitive Debriefing RLS without ADHD 2 3 1 2 2 1 11 RLS + ADHD 2 2 1 1 3 1 10 Total 9 5 7 21 RLS, restless legs syndrome; ADHD, attention-deficit/hyperactivity disorder 18
P-RLS Parent Diary Content Daytime RLS symptom severity Sleep problems due to RLS Daytime tiredness due to RLS Impact of RLS on child s daytime concentration Impact of RLS on completion of homework Emotional impact of RLS on child Impact of RLS on parent/caregiver Child complains about RLS Kick legs Difficulty sitting still Time to fall asleep Bothered him/her when getting to sleep How many times child woke up due to RLS How many times getting up out of bed Hours of sleep lost due to RLS Tired Difficulty staying focused Being distracted by RLS Impact on homework Irritable Grumpy Upset Frustrated Annoyed Parental sleep disturbance Parental frustration P-RLS Parent Diary Content Daytime RLS symptom severity Sleep problems due to RLS Daytime tiredness due to RLS Impact of RLS on child s daytime concentration Impact of RLS on completion of homework Emotional impact of RLS on child Impact of RLS on parent/caregiver Child complains about RLS Kick legs Difficulty sitting still Time to fall asleep Bothered him/her when getting to sleep How many times child woke up due to RLS How many times getting up out of bed Hours of sleep lost due to RLS Tired Difficulty staying focused Being distracted by RLS Impact on homework Irritable Grumpy Upset Frustrated Annoyed Parental sleep disturbance Parental frustration 19
Example Parent Quotes Urge to Move In her leg, she gets I guess she describes it as bugs crawling - feelings in her legs 9 year old girl Yeah, she said that they kind of feel like there's something crawling on her leg - they're kind of tingly. And that she just gets a feeling, and she presses her hands on her legs, and tries to make that feeling stop 16 year old girl Oh, couple of times, two, three times a day, depending on what he s doing. It s mostly when he gets in a relaxed state, like watching a movie or playing a videogame or something like that, the next thing you know I hear him like jump up off the bed or wherever he is and go do something 16 year old boy Uncomfortable Sensations He s described them to me as like a creepy crawly, like a bug. The feeling of bugs crawling on your legs 12 year old boy Example Parent Quotes She lays on her stomach holding on to her pillow, or she will lay on her side so that she can kick her legs - 9 year old girl Kicking Legs Sleep Yeah. He - it just seems like he can't get comfortable, and - you know. He'll - he kicks a lot, to try to get comfortable, or just - because it hurts 8 year old boy Her legs won t settle down enough for her to go to sleep 9 year old girl Usually he wakes up in the middle of the night. It's not usually when he's trying to fall asleep - it's usually the middle of the night 8 year old boy 20
Example Parent Quotes Impact on Homework She still has a hard time making it through a book. She cannot sit still long enough to read a book 7 year old girl Well, a lot of times - like she ll sleep with me But also, she may not be aware of it, but I mean it s like karate chops to your knee 13 year old Impact on Parent s Sleep Importantly there was no substantial differences in the concepts identified or the quotes between ADHD and non- ADHD children Item Development Morning diary with questions focussed on last night Evening diary with questions asking about from when your child got up until he/she went to bed Short recall periods for more accurate recall and to avoid parents having to average over several days Specific and simply worded questions, with each asking about a single concept Clear, simple, non-overlapping response options I don t know response option included in all items pending psychometric validation to identify items parents have difficulty responding 21
Example Items Challenges/Discussion Points Is assessment of concepts that are less concrete/objective appropriate for an ObsRO? e.g. Emotions (grumpy, worry) Tiredness/fatigue Concentration Such concepts are not appropriate as primary endpoints to support regulatory claims, but may still provide valuable information for inclusion as secondary or exploratory endpoints Challenges in practical application should a parent be asked to complete an assessment on days they have only been with the child for a few hours? Can a parent base assessments partly on comments of other adults? Even if they are advised not to will they anyway? Does an emphasis on direct observation result in problematic levels of missing data? Can children who do not always stay with the same parent be included in trials? If parent and child reports are included in a trial which should be made primary? May vary by age 22
Conclusions P-RLS Severity Scale Parent Questionnaire developed to assess signs and key functional impacts based on parent observations Items developed to assess the signs of RLS that are clear, specific and observable But all rely on the parent being with the child and observing closely Expert clinical input critical to ensure clinical relevance of concepts Items focussed on impact daytime concentration, homework, emotional wellbeing are less concrete, more distal, unlikely to be acceptable to regulators to support label claims Could still be of value as exploratory endpoints This ObsRO was developed for use alongside child PRO Could be of value as a primary endpoint, or as a stand-alone measure in children too young to selfreport (e.g. 6-8 year olds) Qualitative results suggest endpoint hierarchy should vary depending on age of child to be verified through quantitative studies Qualitative findings suggested that parents may provide more reliable reports for some concepts, even for children old enough to self report Sleep disturbance Grumpy/cranky Conclusions Measurement issues are complex in the context of dynamic child development Child-report is preferred as direct evidence of treatment benefit Below age 7, child-report is often questionable however; dependent upon concreteness of concept Use verifiable report of observable concepts (not proxy) when child-report not appropriate 23
Particular issues Cannot measure symptoms Figuring out what can be observed reliably and meaningful well defined and reliable If the observer is not with the child all the time cannot mix observers id the reporter Training the observer Conduct of the trial 24