Content Validity: Establishing and Reporting the Evidence in Newly-Developed Patient-Reported Outcome (PRO) Instruments for Medical Product Evaluation - Good Research Practices Donald L. Patrick PhD, MSPH, Laurie B. Burke RPh, MPH, Chad J. Gwaltney PhD, Nancy Kline Leidy PhD, Mona L. Martin RN, MPA, Lena Ring PhD Elizabeth Molsen RN Liaison to ISPOR PRO Task Force Initiatives Task Force established in March 2009. Task Force Background Donald Patrick Built on previous work of a previous Task Force: Evaluating and Documenting the Content Validity of Patient- Reported Outcome Instruments to Support Medical Product Labeling and Promotional Claims in the United States: Report of the ISPOR PRO Existing Instruments & Their Modifications Task Force(Rothman et. al., 2009). Authors represent a range of perspectives on PRO instruments: academia, government, research organization, and industry. More than 50 ISPOR members submitted written comments during 2 rounds of comments, in addition to those received orally during the Atlanta ISPOR presentation. ISPOR PRO GRP Task Force Reports Appreciation to Reviewers The 3 ISPOR PRO GRP reports published in 2009 were downloaded 5,964 times from April 2010 to March 2011. This is an average of166 times per month each. Content Validity of Existing Instruments Changing Modes of Administration Translation & Linguistic Validation We would like to thank all the reviewers of this paper for their time in reviewing and sending helpful comments for these papers. p We addressed each suggestion individually and discussed all as group. Content Validity: Establishing and Reporting the Evidence in Newly-Developed Patient-Reported Outcome (PRO) Instruments for Medical Product Evaluation Good Research Practices Part I: Eliciting Concepts for a New PRO Instrument Part II: Assessing Respondent Understanding 1. Determine the context of use. Paper 1: Eliciting Concepts Five Steps 2. Develop the research protocol for qualitative concept elicitation and analysis. 3. Conduct the concept elicitation interviews and focus groups. 4. Analyze the qualitative data. 5. Document concept development and elicitation methodology and results. 1
Step 1: Determine the context of use Laurie Burke Step 1: Determine the context of use continued Develop a hypothesized disease model based on literature, experts, and patients. Name and define the concept within the context t of a clinical trial end-point model. Select and define the target population including cultural\language groups. Make preliminary decisions on the range of instrument content and structure. Select a theoretical approach to guide qualitative methods for content validity. Develop an hypothesized conceptual framework for the proposed instrument. Clinical Trial Endpoint Model A diagram or description of the hierarchy of hypothesized relationships among all endpoints in a clinical trial Example: Concepts Indication: Reduce asthma exacerbations Endpoints Primary: Asthma Exacerbation Secondary FEV1 Asthma symptom score Context of Use A comprehensive and clear statement that describes the manner and purpose of use and plans for interpretation of a clinical outcome assessment (COA) Concept measured Target claim Target population Figure 2: Example conceptual framework for a PRO evaluating the concept of pain quality Pain quality Deep pain Aching Dull Itchy Intrinsic and extrinsic sources of heterogeneity considered Type of treatment Type of trial (endpoint model) Surface pain Numb Tingling 2
Step 2: Develop research protocol for concept elicitation & analysis Chad Gwaltney Define the target sample characteristics. Clinical and demographic characteristics Sample size/saturation Select the data collection method - focus groups, individual id interviews, both. Determine the setting and location for data collection. Develop the interview guide Draft, pilot, revise. Not a script stimulate discussion Draft, pilot, revise. Step 3: Conduct the concept elicitation interviews & focus groups Mona Martin Obtain IRB approval. Recruit and train sites. Recruit participants; monitor sample characteristics to assure representation. Select and train interviewers. Conduct interviews implement quality control measures. Record or videotape interviews. Transcribe and clean transcripts. Step 4: Analyze the qualitative data Lena Ring Figure 3: From coding framework to coding dictionary Analyze qualitative data according to theoretical approach used. Establish preliminary coding framework; update as data are coded. Establish coding procedures and train coders. Organize data using a qualitative research software. program Assess saturation. Interpret results. Hypothesized Conceptual Framework PRO Instrument Review Target Product Profile Interview Guide Literature Clinical Experts Patient Interview Results CODING FRAMEWORK CODING DICTIONARY Structure Contains all Expanded to codes assigned, Starting add more grouped by Structure for concepts concept, and Organizing expressed by linked to patient Codes patients during language coding process (quotes) All Code Assignments from transcripts Figure 4: Example Saturation Table for Concept Elicitation Interviews Figure 5: Example Multi-Vectored Approach to Understanding Qualitative Data (Example) Symptom Saturation Table CONCEPT CODES Shortness of Breath Difficult to Breathe Not Enough Air Gasping Shallow/Quick Breathing Wheezing Coughing Chest Tightness Chest Pain Dizziness/Lightheadedness NUMBER OF NEW CONCEPT CODES APPEARING IN EACH TRANSCRIPT GROUP % of TOTAL NEW CONCEPT CODES (Total = 10) Group Where Concept First Appeared Group 1 (N=5 transcripts) Group 2 (N=4 transcripts) Group 3 (N=5 transcripts) Group 4 (N=5 transcripts) 7 2 1 0 70% 20% 10% 0% Necessary Coverage (degree of importance to patient) Patient language Relevance of Concept INTERPRETATION & MEANING Presenting the relation of qualitative ti results to concepts, and wording items with the most meaningful language appropriate to measurement design Variability of occurrence Degree of Difficulty to Patient Attribute to Measure Degree of Bother to Patient Essential Coverage (degree of importance clinically and to measurement strategy) 3
Step 5: Document concept development & elicitation methodology & results Donald Patrick Provide target claims and any other context for use. Describe target population. Provide hypothesized and revised disease model and any input from content experts. Provide endpoint model. Provide conceptual framework and revisions made from preliminary to revised. Step 5: Document concept development & elicitation methodology & results continued Provide study methods via protocols and guides. Provide summary of results, including evidence of saturation. Provide transcripts of interviews and focus group. Track origin and derivation of concepts captured in the PRO instrument. Summarize qualitative data. Provide key references. Table 4: Documentation of Concept Elicitation Part II: Assessing Patient Understanding Five Steps 1. Develop items based on findings from concept elicitation. 2. Design cognitive interview process for the planned context of use. 3. Conduct cognitive interviews. 4. Make decisions to revise the PRO instrument. 5. Document cognitive interview results for evaluation of content validity. Step 1: Develop items based on findings from concept elicitation Chad Gwaltney Develop criteria for item development and evaluation. Choose concepts to measure. Select recall period and modes of administration. Step 1: Develop items based on findings from concept elicitation continued Develop items and match to response scales Assess readability Determine order and sequence Draft Instructions 4
Sample item criteria The item captures the concept that is intended. The item is worded in a manner consistent with the expressions used by patients. The item represents a single concept, rather than a multidimensional concept. The content of the item is appropriate for the mode of administration Step 2: Design cognitive interview process for the planned context of use Nancy Kline Leidy Identify population Design cognitive interview ie process Develop Protocol and Cognitive Interview Guide Step 3: Conduct cognitive interviews Mona Martin o Train interviewers o Train subject to think aloud o Use verbal probes o Monitor interview quality o Record and transcribe o Prepare result summaries Step 4: Make decisions to revise the PRO instrument Nancy Kline Leidy Employ an iterative process. Reduce ambiguity in item language. Assess saturation. Balance respondent input with principles of item construction and decisions on conceptual framework. Step 5: Document cognitive interview results for evaluation of content validity Lena Ring Figure 2: Example Item Tracking Matrix Complete Item Tracking Matrix including final item, final response scale, any preliminary domain assignment, description of intent of item, and patient quotes supporting item intent. 5
Next Steps and Manuscript Submission Donald Patrick Submission to Value in Health in June. Questions or comments, please email: donald@u.washington.edu or pro@ispor.org 6